Give your testers a break! Discover how our test automation solution can help your business.

We believe automation should deliver not only better applications, but also be cost-effective and get your new products and features to market quicker. Why not pick our brains about how we do this for our customers?

What We Do

We are a specialist software testing company backed by our proprietary technology.

We complement your current quality program to create a paradigm shift in your application quality.

  • Consulting Advice from Dev to Delivery

    Consulting advice that covers testing strategy development, expert validation and application architecture reviews Learn More

  • Digital Test Practice Development

    Helping you create a scalable and responsive program to help you improve application quality and release speed Learn More

  • Cross-Browser & Cross-Device Test Automation Services

    Bespoke test automation of end-to-end processes that traverse browsers, devices & applications Learn More

  • Testing Tool: Qsome Technology Platform

    Our world-class test planning, management & execution tool built for the digital age with a focus on digital test automation Learn More

Solutions For Today's Software Quality Challenges

  • Today's businesses & consumers interact with your app across multiple devices, so why should your testing be restricted to only a few?

    A robust testing program caters for functional, performance & security testing across all relevant devices. Does your testing program do this?

    Learn More

  • The entire premise of continuous delivery is speed and accuracy in execution. Both of these elements cannot be achieved using traditional testing tools and techniques.

    The Qsome Technology Platform is built to improve quality at speed. The combination of technology and bespoke services allows you to achieve your continuous delivery goals.

    Learn More

  • Today's users take mere seconds to judge an app's user experience. You should give them every reason to rate your app highly.

    Additionally, the speed at which app updates need to be released requires a serious quality program that inlcudes automated testing.

    Learn More

  • Performance optimisation is a dynamic exercise that requires multiple iterations. Its importance is magnified in a digital context where users expect, rather than desire, responsiveness.

    The Qsome Technology Platform allows users to execute load tests using functional test scripts. No extra investment is needed.

    Learn More

  • We have developed proprietary algorithms that enable more relevant test management & enhanced coverage & oversight of the most at-risk processes.

    Our custom-developed dashboard gives your team a conscise and updated view of the riskiest processes & the outcome of their recent test results.

    Learn More

  • Making sense of data is one of today's greatest challenges and potentially a very lucrative opportunities.

    Our ability to conduct intensive data-driven testing at speed will help verify that your Hive SQL queries are behaving as intended.

    Learn More

A Sample of Enterprise Applications Where We Deliver Results

Make no mistake, we're not a body shop. We provide our customers test management, automation and execution. All our testing services are backed by our leading-edge Qsome Technology Platform.

We create bespoke application testing programs to execute manual and automated regression tests. Despite the complexity of these applications, our customers are successful because our tests encompass greater test depth and coverage, at speed.

Some of Our Customers

  • A Global Supply Chain Giant

    Designed & implemented an end-to-end, automated software testing process & systems

  • A Leading Auto Manufacturer

    We helped design & implement a test management & execution strategy for upgrades of the SAP Business Suite

  • A Financial Services Company

    Implemented an industry best-practice testing strategy for a new customer-facing technology product prior to roll-out

  • A Government Digital Transformation Project

    Parallel testing for a new suite of citizens' self-help apps, including geospatial systems

  • A Global Entertainment Industry Giant

    Functional & performance QA for a mobile app serving content to millions of users

  • A Peak Information Technology Industry Body

    Automated functional testing of double-sided ecommerce marketplace for members

Ready to witness the magic?

Contact Us


Australia: 1300 28 44 92

United Kingdom: +44 20 3769 2460

Global: +61 8 8312 1287

India: +91 9845 00 86 96

120 Collins Street, Melbourne, VIC 3000

2 June 2017

The number 1 reason your test automation fails

Test automation should not feel like a nasty video game that feels like it has been created only to smash your spirit and the last vestiges of your team's willpower. Sadly, it often feels like this is the case.

Being at the coalface of software testing, and particularly test automation, we see first hand where the industry is getting it wrong when implementing test automation. Interestingly, for something that should be so intrinsic to modern, digital application delivery programs, most people who fail with test automation the same wrong things.

I think this is because the software testing industry as a whole has not been forced to innovate and, as a result, been forced to adapt to change. So when the "experts" moved from manual testing to automated software testing, they simply applied lipstick to a pig did the same things with a renewed flair. Such practice reminds me of a very famous line that is very commonly misattributed:
Insanity is doing the same thing over and over again and expecting different results.
Said by a very sage person

Why do I know this secret?

First, it's not a secret. It is an unfortunate consequence of a lack of effective and practical knowledge-sharing about how to move from manual testing to automated software testing. We do this from some of the world's largest companies and, frankly, we do it well; we deliver results, so I do know what I'm talking about.

You want proof? Well we recently finished the first round of automation for a customer in Europe where we cut their testing time for a major release down from 4 weeks to 36 hours! What could such results do for you?

Why are you automating your tests?

When discussing test automation with a customer we often rephrase this quesiton to, "do you NEED to automate your software testing?" Let's assume that you have made this decision for the right reasons, you would most likely identify with this finding of the World Quality Report:

If you identify with this significant group of IT decision makers then you are most likely in pursuit of a solution that allows you or application development effort to be more nimble and responsive to the needs of your business. Ideally, automated software testing should enable an application delivery process that helps the company achieve not just its IT-specific goals, but more importantly, it's commercial / business goals.

Check out our Ultimate Guide to Test Automation for IT decision-makers if you want to understand how to map your goals to an efficient automated software testing program that actually delivers results. The Ultimate Guide also includes a great comparison of the various automation engines and popular software testing tools.

However, it is never enough just to have your heart in the right place. Nice guys do win, but not because they're nice! The answer to "why are you automating your tests" may have been your original goal when your team began on its test automation journey. This goal, however, may have changed based on situational factors. Ask yourself, are your tests failing because the goalposts have changed, ie. am I trying to achieve something that I didn't design my test automation program to facilitate?

If your answer is no, then you will likely identify with the benefits of automated software testing that others in your shoes are trying to achieve.

Want to see how our testing tools & services can cut your testing time from weeks to hours?

The expected benefits of automated software testing...

...Are fairly consistent across most industries. The World Quality Report collects them into 6 distinct buckets and we find that our customers generally aim for all 6 of these benefits when partnering with us for test automation services:

We like graphs and numbers so we came up with our own comparison of the benefits of test automation over manual software testing. We've tried this on some of our customers and they agree that the "Green Zone" encapsulates all of the six benefits above:

The Green Zone also attempts to convey that reaping the benefits of test automation is not just about tools or infrastructure or processes or people / service providers. It's about ALL of these things! This is a whole new topic and if you're interested, check out my LinkedIn article where I discuss the confluence of these factors in relation to using test automation to battle the spiralling cost of software testing.

Identification of these benefits is important, because the most common failing of test automation programs is usually a victim of trying to achieve these outcomes. If your anticipated benefits are different and your test automation is not delivering, then talk to us for personalised help.

The essential components of automated software testing

Remember, any of these components in and of themselves are almost worthless. They must work together to achieve the final outcome that is important to you.

None is more important than the other and each element must be carefully calibrated to enable collective successes. The most common mistake that test automation programs make is to assume that automation will reduce their reliance on people. Test automation does not reduce your reliance on people. It increases your reliance on good team members, but it should reduce the overall size of your testing team.

The reliance on smart team members is brought about due to a couple of significant factors:
  • The robots that will perform your test automation need smart people to tell them what they do in the first place; and
  • The time savings through automation should be reemployed to helping you innovate further. For this you need very smart people.
The invaluable outcome of upskilling and redeploying your team to higher value tasks is often a missed benefit of test automation. The benefit is real irrespective of whether you outsource your software testing practice or you keep it in-house.

The interplay between the 3 P's is best explained in our guide to 10x Your Testing Effectiveness for Digital Applications. Download the 10x guide for the latest best practices for software testing. Explaining all that here will turn this blog post into a thesis (and I'll also need a pulpit!).

Need some personalised help about how you can fix your test automation or get it off the ground?

So which component are you getting wrong?

Probably a combination of the three P's. But for a clue, I looked at the following graph from the World Quality Report and it left me highly perplexed:

If things like test data automation, robotics automation and test environment virtualisation are "emerging automation techniques" then our Qsome Technology Platform should have scooped every innovation award in the software quality space for the last few years!

The other USB plug that stood out in that graph (for the right reasons!) was "test design automation." At Qsome when we are automating tests for our customers, this activity incorporates a broad range of activities including, test design, test scope selection and scripting.

However, the one area that we find makes the biggest difference for customers who came to us after failing with their own test automation efforts is in test design. Why? Good question! The most common flaw in test automation efforts is the the practice of writing automated scripts for manual test cases.

What's wrong with that you ask? Another good question. There are two problems with this approach:
  • Manual test cases usually cover well defined functions within your applications. When they cover inter-functional user journeys, the logic of the test becomes cumbersome and unwieldy, causing the automated test to fail at multiple points.
  • Manual test cases are designed to be run once. Maybe 10 times. They are simply not engineered to be run hundreds, or even thousands, of times, causing them to fail often when automated as is.

If you are undertaking quality for a large application then you may literally be shooting yourself in the foot, to the tune of:

How do you fix your test automation program?

First ask yourself, are you automating the right tests? If you are satisfied that you have that right, then look closely at your test design processes, specifically the person undertaking automated test design.

I'm not suggesting you fire them, but inquire as to whether they have the know-how necessary to convert manual test cases into automated user journeys that can be executed hundreds or thousands of times by robots. If your answer is no, then you might need to find someone who can help you this crucial function.

Test automation is not just about speed. It requires intelligence, adaptability and much gumption. When you have those three characteristics in order then you can focus on getting these 10 things right!

If you need help in cutting testing time and finding more bugs before your customers find them or simply to fix your test automation program, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Ayush Trivedi - 11:04

12 May 2017

5 unexpected reasons why test automation outsourcing fails

Failure isn't fatal, but failure to change might be.
John Wooden
And therein lies the crux of the problem with many outsourced test automation services providers. The ability and willingness to acknowledge, analyse and learn from mistakes, both your own and those of others, is a very noble and productive human characteristic, but sadly and uncommon one.

Far too often we receive mayday calls from prospects who have invested a lot of money, but more importantly time, into selecting and working with a outsourced test automation services provider, only to be rewarded with a never-ending tug of war that inflicts much damage to their professional reputations. At the root of these frustrating and poor outcomes is a test automation service provider who was either out of their depth, unwilling to own up to failure and probably did not fully understand the customer's desired future state.

In the case of outsourced test automation services, most customers are trying to prevent losses of the magnitude reported by Computer Weekly magazine from the UK:

One analyst, who follows the outsourced software testing space closely, believes:
Cost savings are a major driver and also the need to do testing in a more professional manner.
Dominique Raviart, analyst at Nelson Hall

While he is definitely on the mark, it is obvious to us that the first desired outcome often drives a disproportionate share of decision-making when selecting outsourced test automation serviced providers. This only serves to diminish the importance of and attention to actually conducting "testing in a more professional manner." In fact, as far back as 2007, Gartner predicted:

Although the statistic above related to the BPO industry but our experience tells us that this would also hold true for outsourced test automation engagements. Gartner research director Alexa Bona put that statistic down to high staff attrition rates suffered by many outsourced service providers. If you think about it, there are no free lunches, right? If you pay peanuts, you get...

While some test automation services outsourcing engagements fail because of technical reasons, it is our analysis and that of other storied industry veterans, that most fail because of misaligned priorities, inadequate discovery and the IT services industry's propensity to try and fit square blocks in round holes.

At Qsome, we ask a lot of questions up front before agreeing to work with a customer. This process of understanding one another is an important step in building confidence and assessing whether the relationship will be one of mutual wins. We have seen that the following five questions are often not asked or not understood during the "courtship" phase between a customer and the test automation services provider. Its almost never as simple as just these questions alone, but they are a great starting point if you are trying to figure out why your test automation services provider just isn't delivering.

The 5 often forgotten reasons outsourced test automation engagements fail are a set of questions that you should ask yourself before your outsource your next engagement. Answering these questions for yourself even if the outsourcing service provider does not ask you these questions will help give any engagement a greater chance of success.

1. What are you trying to achieve?

As an outsourced software testing service provider, this helps us quickly prioritise wins from a potential engagement, as the customer sees them. To the earlier link between the predominance of cost saving motives in failed engagements, answers to this question also help us understand if the customer is looking for fast and effective test automation services that really kick goals, or just an improvement to their cost control target for KPI assessment.

By no means is attempting to achieve cost savings an unworthy goal. However, if the customer is only interested in cost savings then there is little on which to form a lasting and mutually fruitful relationship.

If your software testing outsourcing partner has not asked the overarching goals for engaging, then you should explain these goals and tie progress evaluation to the achievement of these goals. Mutual and detailed understanding of the desired future state is the bedrock on which your software quality will flourish or flounder.

The future state should define quantifiable and measurable metrics covering software quality, internal and external resourcing, timeliness and investment targets.

Want to see how our testing tools & services can cut your testing time from weeks to hours?

2. What are your expectations of us?

The desired future state you described above is only going to be achieved if the combined labour of the customer and the outsourcing service provider produces the desired results. Just because you outsource something, does not mean you are absolved of all responsibility and accountability.

We ask this question to understand two things:
  • How hands-on the customer wants to be throughout the project; and
  • How the customer views their involvement in the project changing over time.
The answers to this question also help to clear many of the unknowns and misconceptions that are very common in outsourcing engagements and lead to increasing distance between both parties. You can minimise, if not eliminate, the "crossed wires" and "we thought you were taking care of that" moments by clearly understanding the roles each partner will play throughout the engagement.

This clarity can be a big boon in engagements where customers want us to develop a test practice for them and leave them to run it. It goes without communication that overall outcomes are significantly improved through this greater degree of communication.

3. How important is this goal in the big scheme of things?

This answer really tells a story about customers' priorities. Are your non-financial priorities at least on par with your cost saving goals? Are you really serious about finding bugs before they get into production or are you more concerned about shifting risk?

Prioritising these goals will help you make a more informed decision about your preferred outsourced testing provider. Some outsourcing partners will be happy to simply help you achieve your cost targets, whereas others may decide that they would rather concentrate on projects where cost reduction is one element of a good outcome.

It is a horses for courses world, so make sure you communicate your goals in prioritised order so that you can attract the outsourced test automation partner that is best aligned to what you are really trying to achieve.

Our customers have reduced bugs in production by over 70%? Want to see how?

4. How will we talk to each other?

Phone, email and Whatsapp are all relevant and useful modes of communication. Many stalwarts of this industry also produce very fancy looking PDF or Excel reports prior to every progress meeting. But, how do you really assess project progress?

To remove this information asymmetry we developed a real-time dashboard as part of Qsome's set of functional test automation tools and load testing tools. This dashboard is not only the central source of truth for project progress, because our test engineers see exactly what our customers see, but it also helps our customers make highly objective decisions about if and when they release to production. This removal of subjectivity between both parties and within both parties is a critical trust-building exercise.

By eliminating any doubts about the veracity of the information being discussed, we enable deeper, more frequent and more productive discussions about how we can achieve the customer's desired future state even faster.

Ask yourself, how could objective and accurate real-time data exchange between you and your outsourced test automation partners improve the results that are delivered by your team for the organisation? Think about it.

5. Have we understood you properly?

The clincher! All the thinking and explaining and strategising you have done to date are of no value if you and your outsourced test automation partner are not on the same page.

A simple technique that we ask our team to practice during sales meetings and progress review meetings is to explain what they think they have understood in their own words back to the customer. Nothing promotes clarity and consistency like repetition.
Any idea, plan, or purpose may be placed in the mind through repetition of thought.
Napoleon Hill
If this question does one thing, it is that it quickly brings unrealistic expectations to the fore. By repeating your goals and expectations aloud back to you, your potential outsourcing partners will be able to quickly decide if they are really attracted to you or if they just want the money. If it is the former, then you may just be on to a good thing.

If you picked up a strong whiff of communication being the central theme behind these questions, then you are on point. Good communication can be an abstract concept to those who miss the forest for the trees. In fact, you know communication is non-existent if someone believes they have communicated well with their outsourced test automation partner because the "legal agreement is watertight!"

Let's be honest, if any commercial relationship devolves into both parties quoting from the contract, you may as well close the door on achieving any goals. While this is an unfortunate fact of life, you can minimise the chances of suffering this fate if you and your test automation outsourcing service provider understand your answers to these key questions.

If you need help in cutting testing time and finding more bugs before your customers find them, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Meena Kumar - 14:24

8 May 2017

How powerful Salesforce test automation can help you win

Most customers who have engaged us for Salesforce test automation can identify with at least one of these issues with manual Salesforce testing:
  • A bad release that has cost lots of revenue, profits or reputations;
  • Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;
  • Apps built by different developers are causing conflicts in production; or
  • Your or your team are copping heat because of poor performance.
In fact most of our Salesforce automated testing customers can also identify with this situation:

This situation is usually the result of stressed testers who are inundated with testing requests that they simply cannot complete with any reasonable level of accuracy or integrity. One consolation point is that this symptom is not unique to your Salesforce testing program alone.

The problem with manual-only Salesforce testing...

...comes down to three primary factors that are common across most manual-only Salesforce testing programs that we have analysed:

  1. Time - comprehensive tests take days if not weeks. Who has that much time?
  2. Coverage - even the most thorough manual testers can't test every nook and cranny in the time available.
  3. Accuracy - with so many moving parts to every customised system, not even the most thorough manual testing program can adequately check for every possible regression that may creep back as the code base grows.
The folks that test the core Salesforce platform internally before it is released to their customers have this to say about their testing practices:
Here at, we use the open-source Selenium project to execute over 40,000 test cases against the features of the Salesforce application.
Salesforce Engineering Team
What's good for the goose has to be good for the gander, right? But what that statement above does not tell you is that Salesforce has built its own testing tool for which it uses Selenium as an automation engine. This allows it to remedy the three problems I described above when running its automated regression tests. I'll tell you how you can get access to a similar tool a bit later.

No matter how much you try a manual-only testing program for Salesforce will result in errors, inefficiencies, bugs creeping through to production and ultimately loss of reputations, for the entire Salesforce delivery team. A steady stream of such occurrences cause end-users to lose faith in the systems, leading to a vicious cycle of inefficiencies, ineffectiveness and the departures of otherwise good and talented professionals.

Size and complexity

Although they are often related, the reason manual-only Salesforce testing programs often end in tears is because even a stock-standard Salesforce implementation is a complex beast. However, most organisations customise their Salesforce environment with third-party apps. While this extended ecosystem helps to deliver a more workable solution for the organisation, this situation becomes a headache for Salesforce testers because the number and interdependence of elements that they have to potentially test goes well beyond the small matter of the 40,000 that the Salesforce engineering team described above.

The interdependence problem is a significant one. It is also one for which manual Salesforce testing techniques have no answer. A study of Salesforce administrators came up with a ranked list of their Salesforce headaches:

You will notice that points 3 to 11 in the graphic above lead to point 12. The complexity of third party integrations, the need for data-driven quality assurance, managing user roles all combine to make Salesforce one incredibly difficult application to test manually.

If you haven't yet understood my point, this additional graphic from the same study should help hammer it home:

Now you might look at this and be forgiven for thinking that testing Salesforce on mobile devices isn't really a problem here because it's such a small fraction of the overall access points. If you came to this conclusion then you couldn't be further from the truth!

Why? Because the people who access your Salesforce environment on mobile devices are usually your colleagues in the field. Salespeople, field workers, managers approving orders, etc. These people are the ones who rely on your Salesforce system to stay to connected with each other and more importantly, stay connected with their stakeholders, who are most likely to be your customers.

Also, that particular study was conducted in 2015. Current access rates of Salesforce systems from mobile devices will likely be many multiples of the figure above. Now tell me that it's ok to not test your Salesforce environment for mobile devices and tablets!

Another report from 2016 found this:
If you belong to the 36% above, then the complexity of your Salesforce system just skyrocketed. Just think about the interdependence of data and processes that now needs to be tested. If you continue to do this manually then two things WILL happen:
  • The costs of testing and time it takes to test your Salesforce ecosystem will blow out as more manual testers will need to be hired; and
  • You will not be able to keep track of all the bugs that need to be fixed or have been fixed or have resurfaced after a fix, leading to very frustrated and angry users who just can't get Salesforce to work for them.
I could go on, as I'm sure could you, but I am confident that you get the drift of why a manual-only Salesforce testing program is ineffective and inadequate.

Want to see how Salesforce test automation could save you weeks every time you test?

So if not manual testing, then what?

Let's be clear, I'm not suggesting that you should cease and desist from manual testing of your Salesforce environment entirely. In fact anyone who tells you to do this is either a pretender or high on something. Manual testing must be used smartly and in concert with a robust and flexible test automation program to really improve the quality of your Salesforce environment.

But be careful, because many of the products that purport to offer Salesforce test automation are really only good at cycling through screens. They don't actually do real regression testing. Many of these products are often considered "industry-standard", which is a reason why many IT decision makers harbour the view that Salesforce test automation is an expensive and ineffective investment. In fact, start to run in the opposite direction if you are told that your Salesforce test automation solution will utilise "record-n-play" technology to build your automated regression suite.

The guide to 10x your testing effectiveness makes this very astute observation, among other very on point best-practice tips:
Record and play does not cut it.

So what does smart Salesforce test automation need?

Any effective test automation program is a combination of three very important factors:
  • People - use SMEs judiciously and at the right times to help your test designers and automation engineers
  • Processes - don't spend time creating process maps, focus on creating robust regression tests
  • Products - select a Salesforce test automation tool that is most suited to what you actually need, not what others think is "the best"
Read this guide to 10x your testing effectiveness to understand how these three factors can be combined to deliver a truly best-practice Salesforce test automation program.

Salesforce, like other complex ERP and CRM systems, has nuances that make test automation difficult. This is why you need to be sure that the people or organisations you select to build and execute your Sales automated regression testing suite have experience in overcoming these nuances. While none of the challenges posed by Salesforce are show-stoppers for test automation, only the best test designers and automation engineers have the know-how to build Salesforce test automation that will not be bogged down by an automated mess.

At Qsome we have built a specific framework that enables faster builds of automated regression tests for Salesforce. It was while building this Salesforce-specific framework that we created functionality to take care of asynchronously loaded elements, particularly within Salesforce's console app. This intimate know-how allows us to deliver value fast and often during our engagements.

Want to see how a framework built for Salesforce test automation could save your bacon?

OK, but how can Salesforce test automation "help me win?"

Winning is a very relative term. The context that helps you decide you need or want to win is just as important as the destination. For some people it's a job well done. For others it is the accomplishment of goals that were previously thought to be impossible within their organisation or among their peers. For other still, it might save their job or win them awards.

Whatever your motivation for implementing Salesforce test automation there are three outcomes that you must demand of your team or your service providers:
  • Cut testing time from days or weeks to mere hours
  • Increase test coverage and the robustness of each test, particularly with negative testing
  • Provide test-related analytics that determines when you should release to production
While most Salesforce users are satisfied with even just one of the above outcomes, you will be settling for a sub-standard outcome and literally leaving money on the table if your Salesforce test automation program does not deliver all three of these outcomes. Without achieving all three of these outcomes you are not maximising the Green Zone:
Read page 4 of our Ultimate Guide to Test Automation for a brief business case of the ROI that's possible from an investment in test automation. The example used in the Ultimate Guide is based on a sizeable, but simple application. As we learned above, the size and complexity of a customised Salesforce environment makes it an application ripe for reaping huge dividends from test automation.

Implementing Salesforce test automation is by no means an easy task. If you need help in figuring out the right plan of action or how to go about implementing your plan, speak to us understand how we might be able to help you:
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Ayush Trivedi - 16:51

3 May 2017

Test automation services for web & mobile applications

So you're on the hunt for test automation services for your web or mobile app, right? But what is it that you need, exactly? Have you defined your requirements or are you willing to be persuaded by the glossiest marketing brochure?

There are thousands of test automation service providers willing to grab every last dollar off you. But there are a few things that you need to be sure about, before signing long term (or even short term) agreements with these service providers that put your professional reputation on the line. In fact, there are 5 key decision points you must consider before you make this decision.

Understand the context

What is it exactly that you're putting at stake with this decision? Is it the quality of applications that allow your customers to spend more money on your goods or services? Is this an application that your team members or vendors use to make sure that you are actually able to service your customers? Essentially, you need to understand the size and ramifications of the opportunity before you can effectively decide on the most suitable test automation services provider.

Most people or organisations in your shoes look for external help with test automation because they identify with at least one of these factors:
  • A bad release that has cost lots of revenue, profits or reputations;
  • Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;
  • Upgrades are long delayed because quality can't be tamed; or
  • Your customers are switching to the competition because your application has too many bugs in production.
We know this because of two reasons, a) our customers told us about these problems when they started using our test automation services, and b) all of my colleagues in our leadership team have faced these issues in our previous lives. Boy, do we have some embarrassing stories on this front!

You are definitely not alone in facing such issues. You're also a step ahead of the competition because you are doing your own research about the best solution for you, rather than jumping into the arms of the first vendor who tells you what you want to hear.

Your decision should be influenced by 6 key factors that go to the heart of every good relationship between an organisation and its test automation services provider.

Testing tools

Do you have existing testing tools that you are mandated to use? Are you prepared for your testing services provider to recommend and use tools that may be a better fit for the web and mobile applications that need to be tested?

The issue of automated software testing tools is central to any such engagement. Choose the wrong tool and you are effectively building many future inefficiencies and frustrations into the project. By selecting the automated testing tool that is best for your needs you will lay the groundwork for all parties to be more effective in performing their role and you will should also be able to reduce costs during the engagement.

Remember, previously popular tools provide the greatest incentives to service providers, but they haven't kept pace with testing needs of fast-paced digital environments that are designed to meet very exacting customer standards. To test digital web and mobile applications you require functional testing and load testing tools that are built for this purpose and will scale with your development needs, without requiring you to blow exorbitant amounts of money just to keep their infrastructure alive and up to date.

Our Ultimate Guide to Test Automation has a comparison of testing tools that you will find helpful. The guide to 10x Your Testing Effectiveness has best-practice tips that you must insist on using during your engagement. Here's a comparison of the most popular testing tools from the Ultimate Guide:

Objectivity in decision making

Presumably you want test automation because you're looking to cut testing time. This is very achievable and indeed something that any self-respecting test automation services company should be able to give you. However, the additional benefit of test automation is that it throws up a number of informative and unimpeachable test-related data that you or your service provider should use when making quality and delivery-related decisions.

For example, how do you currently decide when a version of your application goes into production? Most testing programs usually require a test or delivery manager to make this decision based on their assessment of quality readiness.

However, test automation activities, when performed with the right test automation tools, enables you to build quality-related analytics into your delivery decision making processes. These anayltics make possible the transformation of an inherently subjective, role-oriented decision to an objective, data-driven decision where an individual is never a bottleneck.

Want to see how our test automation tools could save you weeks every time you test?

Engagement models

Engagements with test automation service providers have not always delivered the value that customers wanted and expected. The reasons for this vary, but usually revolve around misaligned priorities, capability gaps, technology gaps and poor communication by all parties. In fact this could be true for all aspects of IT outsourcing engagements!

Time and materials was a great model for the service provider of eras past. It provided an endless flow of cash and constantly shifting goal posts meant that accountability could never really be demanded by the customer. You should avoid getting stuck in this vicious cycle.

We almost only engage with our customers on fixed-fee and fixed-scope engagements, where everyone has skin in the game and the customer's downside risk is contained while the upside can be maximised. Most capable test automation service providers will also offer fixed-fee engagements. If they don't then you need to seriously think about why this may be the case.

Adaptability and integrity

Let's face it: even with the right testing tools and a great engagement model, things will go wrong in a project. Your applications could throw up nuances that will stump even the best automation engineers; user journeys might change after feedback from end-users; there may even be some bugs that are so difficult to fix that timelines suffer blowouts.

The key to overcoming these obstacles is communication and transparency. To inculcate these principles into our dealings with customers, we don't provide weekly or monthly status reports in PDF or Excel. Instead we use the real-time reporting tools in the Qsome Technology Platform to demonstrate and analyse project progress. The fact that this information is shared in real-time and that our test engineers see exactly what our customers see, we are able to build a greater degree of trust with out customers earlier in every project.

However, not everything can be communicated through a dashboard. Sometimes we have to put up our hand and admit to something that we were trying not resulting in the desired outcome. I find that this transparency with the customer encourages greater sharing of ideas and solutions, which in-turn delivers greater value to the customer.

You must assess whether your test automation services provider is willing to be honest about the project's progress, even when the going gets tough. This is the difference between a truly successful customer-vendor partnership and one that produces mediocre, or even sub-standard, results.

So what do you actually need?

This is the most important question after all, but you cannot answer it without completely responding to the 6 decision points above. There are really only three types of services that any test automation services provider can offer you. The exact service you need right now depends on how you have responded to the decision points above.
  1. Advice or consulting: a root and branch review of your process from development to delivery. This exercise is often useful prior to engaging a test automation services provider because it allows you to better understand the other elements of the puzzle that can feed into making your test automation efforts a success.
  2. Test practice development: this is where your service provider builds you a fully functioning software quality program right from processes, to people, to technology, to actually designing, automating, executing and analysing tests. These services can be a combination of in-house and external elements depending on your goals and priorities.
  3. Test automation services: I call this a "build and let us run" approach where the test automation service provider develops the automated regression suite and leaves it to the customer to execute as and when required. There may be some ongoing commitment from the services provider for maintenance of the regression suite, with a very limited scope.
Download the Ultimate Guide to Test Automation to better understand how you can implement a test automation program that slashes testing time and improves software quality. If you prefer talking to reading, speak to us about your goals and challenges:
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Ayush Trivedi - 14:00

1 May 2017

Powerful automated software testing metrics & why you need them

Automated software testing metrics by themselves are not powerful. It is the smart use of these metrics that makes them powerful and gives you the ability to transform your software quality program. Unlike Qsome's test automation tools, most testing tools do not offer test-related analytics. The analytics that are built into our automated testing tools are one of the differentiators that sets us apart from our competitors.

Elfriede Dustin's book, Implementing Automated Software Testing, is considered gospel for those wishing to put in place a set of metrics to assess the effectiveness of their automated testing program. However, the pace of change in software quality requirements and new development methodologies have necessitated a need to prioritise some automated software testing metrics over others.

At Qsome, there are two types of metrics that help convey very different messages:
  • Metrics to use to assess automated test preparation; and
  • Metrics to use during test execution to help with delivery-related decision making.

It's important to be SMART

Whatever metrics you choose to follow as part of your automated testing program, apply the SMART principle to make sure that you don't waste your time. Applying this elegant principle helps to reduce errors and forces your team to focus on only those metrics that add real value to the analysis of your testing program.

I find that the part of this acronym that people most struggle with is "time-bound". In the metrics that we track with our testing tools, time-bound takes two forms:
  1. The time period over which that activity is being tracked; and (more importantly)
  2. A comparison of the result over different time periods.
Applying point 2 to all analyses of your metrics will help you quickly understand where your automated testing program is ripe for optimisation. This is where automated software testing statistics go from mere metrics to those of the powerful variety.

Metrics to assess test preparation & test worthiness

As the guide to 10x testing effectiveness emphasises, it is very ineffective if manual test cases are simply converted to automated test cases. To work out Percent Automatable make sure you convert your test cases into user journeys before calculating how many can be automated.

While it may be possible to automate every journey, it many not be the most effective or efficient use of resources to do this. My teams aim for a percent automatable metric of over 90%, especially for web and mobile app testing. Depending on your application architecture, it may be excessively cumbersome and unnecessary to automate the testing of setup or initialisation journeys.

Automation Progress refers to the number of user journeys that have been automated, from those that were deemed automatable at the outset. This is an important number for customers who want to understand how their automated testing service provider is progressing with building the automated regression testing suite.

Want to see how our test automation tools could help you release to production ONLY when your application is ready?

Metrics to assess test progress and success

The 10x testing effectiveness guide focuses a lot on how to make your software testing program faster and more effective at catching bugs before they get to production. Increasing the effectiveness of anything, particularly software testing programs requires participants to conduct a series of small experiments, track key metrics and iterate after changing one or two elements to assess the impact of the said changes.

Now that you know what to do with these test automation metrics, the first one to track is very simple, but very few teams actually track it: number of defects in production. This overarching statistic should be tracked for every version that is released to production. Plot it on a graph and if you see a downward trend, then you know that your software testing program is actually working. If your graph resembles more of the latter parts of a J-curve or has flat-lined, then you need help.

The percentage of defects resolved is a very simple statistic that tells you the workload remaining in a sprint before the application needs to be tested again prior to release. Ideally that number should be zero before an version is released to production. It's not rocket science!

An advanced version of these two metrics is defect density. This is the percentage of all defects contributed by each module that is being tested. Because we create automated tests based on user stories, this can be a little more difficult to measure, but it will provide great insights into which parts of your development process may require greater emphasis on quality.

Two metrics that we ask our customers to focus on prior to any release are percentage of tests passed and test coverage. These two are quite self-explanatory, but must form the basis on which all application delivery decisions are made. If both are not above a pre-agreed threshold, that release does not make it into production. Quite simple really!

For large applications like ERP and CRM systems, test coverage can be split into the coverage of high and medium-risk user journeys or coverage of only those user-journeys that that are affected by any code changes made in the current sprint. To accomplish the latter you require your automated software testing tool to be able to automate test scope selection.

The list of metrics that can be tracked is literally endless. The question you have to ask yourself, is whether it is necessary for other metrics to be tracked within your software testing program.

How will you track these automated software testing metrics?

Knowing what to track is one thing, but executing the measurement process without burdening your software testing program with loads of extra resources, inefficiencies and costs is also important. Ask your testing tool vendor what metric tracking capability is already built in to the tool.

You will likely find that very few automated software testing tools have metric tracking included out-of-the-box. Qsome's testing tools do include critical test analytics to drive the software delivery process. This is one of the reasons that our customers have switched to us.

Download the 10x your testing effectiveness guide to understand how you can slash testing time and improve your software quality. If you prefer talking to reading, speak to us about your goals and challenges:
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Ayush Trivedi - 17:56

26 April 2017

Which is the best automated software testing tool for my organisation?

If you were expecting a ranked list of the best automated software testing tools, which is most likely heavily adulterated by promotional payments, you have come to the wrong place. Instead of giving you a ranking of tools, I will show you how to pick the testing tool that is the best fit for your purpose.

Unfortunately, unlike Tina Turner's epic song, there is no one testing tool that is "simply the best." Selecting a automated software testing tool is like choosing a life partner: your ex may not have been great for you, but they could be perfect for someone else. Just as we subconsciously select our spouses based on data points like personality, humour, looks, you get the drift, there are 7 elements of every software testing tool that you must consider before deciding which tool is best for you and your applications ecosystem.

Software testing tools appear to be a big issue as far as implementing highly effective testing programs are concerned. The World Quality Report 2016-17 reports that 45% of don't believe you have the right testing tools, especially for mobile and multi-channel testing programs:

It then makes sense that you are reading this now! We cover these topics in greater detail in our guide to 10x your testing effectiveness. But, in short, you need to consider 7 decision points before you select your testing tool of choice.

The non-negotiables

We always say that the best automated software testing tools include the some critical features, without exception:
All good software testing tools include test management, planning and execution. Only the best software testing tools also include automated test execution, automated scope selection and comprehensive debugging tools and analytics too.
Gone are the days where your test management, test execution and defect management can be spread across multiple tools. Disregard any software testing tool that does not have all of these features in one place. If you don't, then your team will be stuck in an arduous cycle where they spend more time learning and taming their testing tool, than actually using it to their advantage.

What are goals do you want your automated software testing program to achieve?

People who set goals and write them down make 10x more money than those who don't. Apply this theory and set your goals before launching your quest to find the best automated testing tool.

Unlike setting life goals, setting goals that help you select the right testing tool only requires you to answer one question: what is that that you want to achieve through a new testing tool? Is it:
  • Reducing the time it takes to test?
  • Making it possible for team members with no testing experience to execute automated tests?
  • Giving your developers videos and screenshots of bugs that are found during tests?
  • Using objective data to improve decision making about when to release apps?
  • Making sure bugs are found before your customers find them when using your application?
  • Being able to test a process across different applications and browsers in one test?
  • The need for a tool that doesn't needs weeks of expensive training before your team can use it?
  • The ability to conduct load tests using only your functional test scripts, with no effort duplication?

Answers to these questions will get you in a better headspace to accurately deduce which features of an automated testing tool will make a real difference in your organisation. In fact ranking these answers in a prioritised order will make the decision even clearer for you. However, without these answers, you leave yourself open to being easily influenced by meaningless sales-speak, upsells and cross-sells that will leave you even more confused.

Who will use your testing tool?

This is a question of resourcing. Do you have or want to employ a pool of highly proficient test designers and test automation engineers? Or would you rather outsource the development and management of automated tests while having non-testing users who are part of your team actually execute the tests?

There is no right answer to this question. But your answer will make a difference to the types of tools you can realistically invest in for your software quality program. Most popular testing tools require expensive resources and specialist skills just to operate them. Our customers enjoy a different experience where their non-test users can execute a batch of automated tests, analyse the results and assign defects to developers using our functional test automation tool and load testing tool. There's a lot to be said for a test automation tool that does not require weeks of training.

The other issue, often spoken about in hushed tones, with maintaining a specialist testing team is that good testers are hard to recruit and even harder to retain. What is the value proposition of your testing program that will encourage and compel a team of high quality testers to buck this trend?

Want to see how our test automation tools could save you weeks every time you test?

What does your applications ecosystem look like?

Do you have a few custom-build Windows desktop apps or is your company well and truly on its digital transformation journey? If you're going digital what part do mobile apps play in your applications ecosystem? If you do have digital applications that are used by thousands or even hundreds of people, do you need to optimise them for performance?

Assuring the quality of web and mobile applications that interact with other applications requires more than just test automation. It requires your automated software testing tool to include automated scope selection and risk-based testing. Such applications ecoystems also require a different level of interaction between the business, the test team and the development team. Download our guide to 10x the effectiveness of your testing program to understand how this needs to work.

Keep in mind that very few software testing tools allow you to test a user journey that travels across applications, browsers and mobile devices in one integrated test. If this is something you need to achieve the goals you identified above, then you need a testing tool that is built for digital.

You SHOULD want defect tracking and debugging capability in one place

In this SaaS age most developers and testers are used to using multiple tools to complete a task. Just because they are used to it, doesn't mean it is the right or best way to get the job done. In fact, the most common question we get during our sales discussions is, "I do my bug tracking in Jira, can Qsome integrate with Jira?" Our response, "yes, but how will that help you?"

What follows is usually a realisation that such a set up will make life even harder for their teams. The integration question above usually stems from the school of thought that says don't fix what ain't broken. While I would generally agree with that thinking, you have to really assess the impact of splitting up your bug tracking software with your test management, test planning and test execution tool.

Apart from the intuitive and user-friendliness of our testing tools, one of the things customers really like is the ability to send a defect, including video logs, screenshots and technical results, to the right developers with only two clicks. No need for emails or manual "please fix this" annotations. Your automated testing tool should make your life easier, allow it to do its job so you can do more with your time.

Do you have the capability to set up and maintain your automated testing tool or do you need something that just works?

This is an extension of the resourcing issue I wrote about earlier. But the fact that you're looking to invest in an automated software testing tool is that you want to maximise the green zone, right?
That green zone of benefit is constantly under pressure from a few factors that creep into every software testing program:
  • Using multiple Excel files to configure and execute automated testing scripts
  • Spending endless time and money just to maintain your testing tool and the infrastructure it needs
  • Ensuring that the required results of each test matches the data set it uses
  • Training required just to learn how to operate and maintain the testing tool
  • Ensuring a stable integration with your continuous delivery processes

One of the ways of protecting your green zone is by exploiting the cloud. Cloud-based software testing tools allow you to do away with much of the headache of maintaining expensive testing infrastructure for your testing tool. Do you want a tool that maintains itself and just works? Then you will like this.

Upfront licenses or monthly subscriptions on tap?

Spending half a million dollars to set up test infrastructure with onerous licensing terms was not unheard of for even small teams using yesteryear's testing tools. It makes infinite sense to us to take advantage of subscription-based pricing models where you can dial up or dial down the number of users that need access as and when they need that access.

In fact our Ultimate Guide to Test Automation recommends that you should ask for a proof of concept before you commit to a long term agreement to use a testing tool. After all, isn't software testing about minimising the downside while preparing for the upside?

You read all the way to the bottom because you have decided that automated software testing is going to be an integral part of your future software testing strategy. The selection of an automated testing tool to bring this strategy to life is key to your success. Download the Ultimate Guide to gain a detailed understanding of your key decision points. If you prefer talking to reading, speak to us about your goals and challenges:
Or contact us on +61 8 8312 1287 or solutions[at]
Publisher: Meena Kumar - 15:36