One file example to run data driven tests using python and nose with XML and HTML reports

I was discussing data driven tests with a colleague and how easy it to create in python just with nose test framework.

The file looks like:

def test_dirs():
    for data in test_data():
        yield check_dir, data[0], data[1]

def check_dir(dir_ref, dir_new_gen):
    assert len(dir_ref) > 1
    assert len(dir_new_gen) > 1

def test_data():
    return [["dir1_ref", "dir1_gen"], ["dir2_ref", "dir2_gen"]]

I created a repository  and shared with him following is current content of the Readme:

https://github.com/mubbashir/py_nosetest_data_driven

One file example to run data driven tests using python and nose tests with XML and HTML reports

Documents

• nose test
• Test-generators
• nose-html-output

Optional VENV (Python virtual environment)

• Create virtual environment python -m venv venv
• Activate virtual env
  • Linux/Mac source venv/bin/activate

$ source venv/bin/activate
(venv) [akhan@seq-akh-fc pipeline_workflow_tests] which python
~/workspace/pipeline_workflow_tests/venv/bin/python

Install and execute

• Install dependencies: pip install -r requirements.txt
• Run tests

$nosetests -v
nose_test_generator.test_dirs('dir1_ref', 'dir1_gen') ... ok
nose_test_generator.test_dirs('dir2_ref', 'dir2_gen') ... ok
nose_test_generator.test_data ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.003s

OK

Run with reports

$nosetests -v --with-xunit --xunit-file=nose_test_results.xml --with-html-output --html-out-file=nose_test_results.html
nose_test_generator.test_dirs('dir1_ref', 'dir1_gen') ... ok
nose_test_generator.test_dirs('dir2_ref', 'dir2_gen') ... ok
nose_test_generator.test_data ... ok

----------------------------------------------------------------------
XML: /home/akhan/workspace/py_nosetest_data_driven/nose_test_results.xml
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

• XML Report
• HTML Report

Repository: https://github.com/mubbashir/py_nosetest_data_driven

Gradle task to check files/commands in PATH environment variable before running tests

Recently I was debugging a failing tests and it turn out the when Jenkins runs the tests on a remote node shell environment is not initialized hence one of binary dependency of the application under tests was missing.

Solution:
Add the additional location in the PATH from jenkins
e.g. export PATH=$PATH:/opt/bin/

Force checking in gradle:
In the light of principle like ""treat test code with the same level of care as production code" and To help my fellow tester (and my future self) it's better to add this as check in gradle.

// rest of your build
// Verfied with gradle 4.10
task("checkEnv"){
    doFirst {
        def listOfFileToCheckInPath = ['find', 'grep']
        listOfFileToCheckInPath.each { file ->
            if(!isFoundInPath(file))
                throw new GradleException("${file} was not found in any of the folder in PATH: ${System.getenv('PATH').split(File.pathSeparator)}")
        }
    }
}
/**
 * Static function to verify if a file/command exist in PATH environment 
 * @param file
 * @return true if found, else false
 */
def static isFoundInPath( file){
    def PATH_ENV = System.getenv('PATH')
    def fileFound = PATH_ENV.split(File.pathSeparator).find{ folder ->
        println("Looking for ${file} in ${folder}")
        if (Paths.get( "${folder}${File.separator}${file}").toFile().exists()){
            println("Found ${file} in ${folder}")
            return true
        }
    }
    return fileFound
}
// Making test task to depend on checkEnv
test.dependsOn checkEnv

The output would look something like:

./gradlew checkEnv
> Task :checkEnv
Looking for find in /usr/local/bin
Looking for find in /usr/bin
Found find in /usr/bin
Looking for grep in /usr/local/bin
Looking for grep in /usr/bin
Found grep in /usr/bin

BUILD SUCCESSFUL in 0s

Gist on github

Selenium, Jbehave , gradle and Serenity - Perfect combination for automating BDD styled checks in Java

I have just created this demo project using Selenium, gradle, Jbehave and Serenity

https://github.com/mubbashir/jbehave-selenium-bdd

Why add Serenity?

Serenity adds plugins to Jbehave and makes it a lot easier to write acceptance tests as mentioned on it site Serenity BDD helps you write cleaner and more maintainable automated acceptance and regression tests faster.
The other things I loved about Serenity is the it has wonderful reports  for BDD "Serenity also uses the test results to produce illustrated, narrative reports that document and describe what your application does and how it works."
See:

Structure

├── ReadMe.md  ## ReadME file of cource ;)
├── build.gradle ## gradle build file 
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew ## grdadle warapper
├── src
│   └── test
│       ├── java
│       │   └── test
│       │       ├── WebTest.java ## This is where the tests resides which add all the steps using net.serenitybdd.jbehave.SerenityStepFactory
│       │       └── selenium
│       │           ├── page ## This where all the pages extending net.serenitybdd.core.pages.PageObject
│       │           │   ├── google
│       │           │   │   └── GoogleLanding.java
│       │           │   └── interfaces
│       │           │       └── Landing.java
│       │           └── steps ## Pages are used in steps 
│       │               └── GoogleSteps.java
│       └── resources
│           └── stories
│               └── google
│                   ├── Search2.story
│                   └── search.story

Refrences

• [jbehave]: http://jbehave.org/
• [gradle]: http://gradle.org/
• [serenity]: http://thucydides.info/docs/serenity/#first-steps

My Meeting with James Bach

Last Saturday I had the privileged to meet with James Bach (If you are a software tester, I expect you to know him) and as you can see below I was

Can't express how excited I am :) and he played the dice game with me. Thank a lot @jamesmarcusbach for your time. pic.twitter.com/ZPwD0MfCll

— mubbashir (@mubbashir) September 12, 2015

I asked him about Sapient Testing->Non Scripted Testing->Testing vs Checking  and how these 3 are different.

He corrected my understanding of Oracles I was thinking about it too specific from the perspective of checks by focusing on pre defined rules whereas "In testing, an “oracle” is a way to recognise a problem that appears during testing."

 We talk about check automation where he told me about this wonderful analogy that think of automated checks as a web and tester as Spider.

The spider is responsible for creating it, managing it, to find the right place and spot for it, a place where there is a greater chance to catch bugs and so on...

And then comes the best part when he played the Dice game with me 

Just after few minutes in the game I felt myself bit overwhelmed :)

Few of the points I took as a take away after that session:

• Taking clear and correct (Yes correct) is extremely important
• Simplify problem: Start with the very basic try understand the system with the very basic data
• Don't get yourself too attached with the basic data, once you are sure how it is working for simple data - "Explore", "Randomise" and Look "Below the surface"
• Use Focus / Defocus heuristic when in doubt use focus when frustrated Defocus, walk around, change your perspective of viewing 

While we were walking out of the coffee shop he saw me looking at his Cap (those who know me, would know why :-D I wear Cap all the time ) and then he told me he give Cap to tester who work for.

I have been told by @jamesmarcusbach that only @Lalitbhamare @huibschoots @CuriousTester and @testertested have His branded Cap #SoJealous

— mubbashir (@mubbashir) September 15, 2015

That got some responses with respect to who else has one :D

One thing which made me quite happy was that he was aware of what we Software Testers from Pakistan are doing at the moment mostly because of my dear friend @arslan0644 

I told him briefly about STEP (Software Testers Engagement Programme) an initiative which bunch of my fellow testers took to bring closer the community

@arslan0644 I need to tell you all about my meeting with @jamesmarcusbach https://t.co/HbNsSg5hkq

— mubbashir (@mubbashir) September 12, 2015

Deleting facebook test users of your app

- Create a new user (We know how to do it, sending the key is sensitive so not adding here)

- You will get a response like:

{

• id: "137616849955555",
• access_token:"CAAAAJ6uP3yMBAC56lfzUlZB3uwWms0x6xXoriPn3ZAVko185KZCTxQaYADMqkXhpIH1VxAJtXzJBoNR7TNWKpRBoEAC79AY1VaGss93nHiqVGeZBEvHZALouNw55PfZBujt1Up1PbjEryWB4iptNPF3PDAQuZAup4jGGRFUJeEs6JBvIEpoBvCQZCuB467oznCoE41zDbxFaVGnURP1JdV9H",
• login_url: "https://developers.facebook.com/checkpoint/test-user-login/137616849955555/",
• email: "name_1441262612@tfbnw.net",
• password: "2073387096"

}

- Using the information from the above we can delete the user via the following:

https://graph.facebook.com/137616849955555?method=delete&access_token=CAAAAJ6uP3yMBAC56lfzUlZB3uwWms0x6xXoriPn3ZAVko185KZCTxQaYADMqkXhpIH1VxAJtXzJBoNR7TNWKpRBoEAC79AY1VaGss93nHiqVGeZBEvHZALouNw55PfZBujt1Up1PbjEryWB4iptNPF3PDAQuZAup4jGGRFUJeEs6JBvIEpoBvCQZCuB467oznCoE41zDbxFaVGnURP1JdV9H 

Update the above URL by replacing TEST_USER_ID with the value of id from the first response and TEST_USER_ACCESS_TOKEN with the value of access_token from the first response

https://graph.facebook.com/TEST_USER_ID?method=delete&access_token=TEST_USER_ACCESS_TOKEN

The response from it would be true.

Delete the test users once we are done. 

What is the most important skill a software tester should have?

What is the most important skill a software tester should have?

In the below video are the answers from a conversation with Tony Bruce at EuroSTAR 2011 last November.

Following are the answers:

0:33  Communication- Fred Beringer @fredberinger

1:14 Ability to Think

1:30 Analytical and Technical 

1:50 Objective look, think(+1) and present his/her thought - Fiona Charles @FionaCCharles

2:20 Communication (+1) 

2:30 Communication (+2) 

3:20 (depends) Critical and Through (for some domain)

3:32 Passion and Enthusiasm  

3:48 Knowing what battles to pick

4:14 To learn - (Andy Glover @cartoontester)

4:43 Has to have many (Curiosity, Intelligence, desire for better software/world  )

5: 25 Question every thing

5:32 Think out side the box (+2)

5:46 Think (+3) - Nancy Kelln @nkelln

6:00 Communication (+3) - John Stevenson @steveo1967

6:12 Not an agile tester (coz it has become a hype)- Stefaan Luckermans @stefaanl

(please let me know if you can identify anyone else or for incorrect identification(s) :D)

Thinking and Communication are the winner.

I believe testers should have technical skills, and thinking is pretty much required in any technical job (O.K. in almost any technical job) but see how many people are saying "Communication".

Communication is important, it's real important for a good tester and doesn't only include the formal, template filling kind of communication i.e. Test Plan, Test Cases, Session Reports, Bug Reports or any  other mode of sharing your findings, but informal kind of communication is equally important.

The informal chit chat with a developer or any other team member about the product, process bug or bug fix helps us a lot. Similarly how to be team member rather then acting like quality police, how to help other team members and how to seek help from other team members (yes any team member with any specific role, developers, DBAs, System Administrators included). 

Similarly we should be able to communicate with other testers to learn from them and we should be able to share our views and opinions about testing with other testers.

No matter how good we are, we can not work in isolation.

Server side monitoring in JMeter with JMeter Plugins

Like may I needed some server side statistics while running load tests earlier I had couple of Ruby scripts which were gathering my required information on server as CSV file I do have webmin configured on the server but I was still looking for something which is bit more tightly coupled with one of the most famous (easy to use) tool for load testing JMeter.

Here is how to do it:
We just needed one thing JMeter Plugins
 Prerequisite: JMeter 2.4+ or above with a JRE 1.6+
 Download JMeter Plugins 

• Unzip it and (refer) 
• Copy JMeterPlugins.jar to  JMETER_HOME/lib/ext. 
• Copy zip files content to the Server(s) you need to monitor
• On each server run JMeterPlugins-HOME/serverAgent/startAgent.sh PORT_NUMBER (refer)
• Run JMeter (create your test plan as usual)
• You will see bunch of extra items in Add context menue's item beginning with jp@..  these are the extra plugins we just added via JMeterPlugins.jar
• Add jp@gc - PerfMon Metrics Collector (refer)
• Configure your parameters (refer)

• Run your tests and see server side info

I have use it on Mac and Linux, it should be able to work with most of the systems since its server side agent is build on top of SIGAR - System Information Gatherer And Reporter

Sikuli on Selenium- A demonstration of automation using selenium and Sikuli (such as flash uploader)

Selenium 2.0 is just launched, and the excitement remind me that there are tons of selenium related things I need to share with the world.

One of those things is File Upload. File uploads have always been a real pain to automate using selenium.

People have used AutoIt (so have I) to upload on windows only environment. On windows, beside AutoIt, native events  (of webdriver) have worked a great deal for me in the past. 

It becomes a real challenge  when file upload is done via some flash components (components which don't provides hooks from java script to play with files to upload).  

I had tired couple of things but the thing which worked really well was to have type="file"  and send keys but flash uploader and *nix remains challenging.  

Then I stumble across Sikuli. From there web site "Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots)".

Below is what I have done to use it to automat file upload via flash components on MacOS X. With few image changes we should be able to port it for Linux.

Source code is available at: https://github.com/mubbashir/Sikuli-on-Selenium

See it pretty smoothly uploads the file, but .. It works like a real user and real human user i.e. a real user can only interact with what is being displayed on the Screen. If the browser window dosen't have the focus Sikuli will not be able to find images.

Did I mention that test are written using TestNG, so this can slightly work as demonstration of TestNG on Selenium as well. 

Note: I haven't tried this in head-less mode.

It is Fixed, or Just Fixed

The first thing you tell a Tester is "How to write Test Cases", Then you tell him how to write the Better one, and How can you make them Effective. Then you throw IEEE guide lines on Tester to make even Lengthy. Even the gurus have told us what is a good Test Case.

And Once I follow all these guide lines (not matter I want to follow or not) and find some Bug then you kill me by Telling me how to write bugs and how to report bugs how to use the Bug Tracking system and how to use it Effectively. Even the gurus have told us about writing Effective Bug reports.

And few Sr Tester/ Test Manager believes, always write how to replicate the bug and If you can't replicate its not a Bug.

Asking the tester to follow this (instead of letting him Test) is done in the Name of Communication. Communication with Developers, communication with Product owners and Communication with rest of the Team. And then there comes the advocacy part.

Once I have spent this much time on ( things helping ) Testing, I follow the guide lines for creating Test Case, I follow the Guide Line of Bug Reporting and I advocate my Bug well. Then the developer acknowledge the bug and submits it patch and changes the State in the Bug Tracking System (or an Email thread) and puts his comments. "It is Fixed" or Just "Fixed".

Ohh Come on, I deserve more then that, show some courtesy You ask me to write and write and your response is "Fixed". You think I don't need to know

What was causing the Bug?
How have you fixed it?
What other Areas might have effected with the fix?
(during the earlier cycles of Testing) You don't want me to tell that you have been real Lame in deploying the application (For Testing) you haven't updated the config file you haven't deployed all components, you were using wrong code from wrong Source Code Tag.
You don't want me to tell you can't use regular Expression to handle Input fields.
Or there was a logical Error, You were Assuming it is to be done some other way (and I wan't to know the other way)
......

Knowing the application help me a lot to Test it, What was causing it help me a lot to Understand the Bug Pattern and Help to find similar things in other parts of the Application. What you have done to fix it Help me a lot to investigate Impact of the Fix and How can I make sure it hasn't effected any other area...

Why Test Managers Keep Telling his Testers How to write Test Case/Bug Report and Its dos and don't. Why Can't They Developers to shade some light on the Bug and Its fix.

Why Development Manager Can't Tell developers to write few lines when changing the state of the Bug.

Why It is just "Fixed"?

BTW, you can avoid all this, but for that you need to have good relations with developers you can get all that information on Informal communication channels.

Google Test Automation Conference 2009 - Zurich, Switzerland

Google Testing Blog: Google Test Automation Conference 2009 - Zurich, Switzerland

The word is out, We will be having my all time favorite conference i.e GTAC on October 21 and 22 at the Google offices in Zurich, Switzerland.

I am sure this time we gonna create some noise on twitter with #gtac.

P@SHA ICT 2008 Awards: I am so happy

We just had P@SHA ICT Awards 2008, I am extremely happy with the outcomes. That was a one cool event, The event was covered by CIO Pakistan, @DrAwab And @Selfexile were twitting the event as it happens and a Live Blog was hosted at Ideas Hut website (futher details ). I am so happy with the event :-).

Why I am this }---||---{ Happy?


Well there are two reasons
1. The company which I have just started working for, got the award for "Best of Digital Media & Entertainment Applications" (WINNER - Converged Media Solutions – PixSense ). But this ain't the primary reason of my happiness (They are an excellent company and were bound to win this :) ), so the the primary reason is reason no 2. which is:
2. The award for Best of Startup Applications. I am a software Testing professional and award was won by "Kraysis" which is to the best of my knowledge the only company in Pakistan, specialized in all kind of Software Testing solution and yes they are the partner of Zephyr in Pakistan.

I posted a Blog about Testing market in Pakistan on 14th august 2007 . As a software tester I know this is what we really miss in Pakistan, companies providing Testing Solutions.

There are bazillion companies around the globe which provide software testing solutions but there aren't many in Pakistan. I am extremely happy atleast there is a company now (actually I am following them for more then a year see my blink ) providing software testing solutions and they are doing great.

They were named in the STARTUP Watch-List by UK based Technomics International, they have partnership with Coverity Inc and Zephyr and yes they have won the award of Best Startup (source: http://www.kraysis.com/Deatilednews%20.php ). As a Software Tester from Pakistan I am so happy to see them grow.

But.. They still miss one thing the really need to mobilize the local Software Testing community. I mean they should arrange workshops, conferences, meetups and all those cool things. I would love to see conferences like GTAC or AST Conferences happing in Pakistan and I would love to play my role in them.

With startup like Kraysis and there growth over the year or so, I am very hopeful about the Software Testing community of Pakistan.

Update: Even more happier :-), both of the above mentioned have won awards at APICTA in Jakarta. PixSense has won the Asia Pacific ICT Award and Kraysis has won the APICTA Award for the Best Startup in the Asia/Pacific region.
Further details.

Fix Firefox "Unresponsive Script" warnings

Recently i was facing this problem while writing some Automated Web Functional testing using selenium that firefox pops a warning for unresponsive script (for some how i had to use Thread.sleep(1000)).

This is not only annoying but also hearts the efforts for producing unattended scripts.

Google again come to rescue and i got this from. So the solution goes like:
Go the hidden configuration page in Firefox by typing about:config in the address bar . (make sure that you are doing this for the profile you are using for selenium)
In the 'Filter' box, type script_run_time. This will narrow the options to dom.max_script_run_time and dom.max_chrome_script_run_time.

Right-click it and choose Modify. A box pops up. Change the number to something bigger like 40. This is the maximum time a script can run before Firefox considers it 'unresponsive'.

If you can’t find the string in the about:config page, create it by right-clicking anywhere and then choose New—> Integer and enter there name.

Testing market in Pakistan.

at http://club.drivenqa.com/ Ahsan Choudry asked me about the Testing market in Pakistan.
(update: club.drivenqa.com now have its own domain this thread can be seen here)
Below is my reply to him on this:

Before sharing my point of view I really want to know how do you see it?
Well it is growing as now organizations are looking for Software Tester. I am doing Software Testing for around Two years now, and I know some great developers in Pakistan but unfortunately i haven't came across many Testers. I am pretty sure that there will be(or there should be) software testers in every organization which develop softwares but i don't see them around on line and a simple googling can reveal it .
But there are people who are doing software testing see the google trends, we(The Testers of Pakistan) are just not verbal enough!
(Update Pakistan Rank 2 by that time)
for me the reason for this is we do not have many institutions teaching Software Testing. When your words are not backed up by your education you wait for you experience to back them up. I am pretty sure you will see more and more Pakistani Testers online pretty soon as our experience will become our Teacher..
The Bottom line is Market is growing, Organizations have Software Testers, They are hiring more and more Testers but we(The Testers) haven't started claiming our cyberspace yet.