How to select a provider of automated software testing as a service

The selection of the best and most compatible partner to provide automated software testing as a service could be the difference between your applications delivering real business benefits or turning into a damp squib. The positive or negative results of this decision will be magnified because of the speed and nuances of automated software testing.

We strongly recommend a thorough perusal of Ultimate Guide to Automated Software Testing to help your understanding of the groundwork that needs to be done before selecting an automated software testing solution provider. Once this preparatory work has been done, there are six key questions points that you must answer before partnering.

Price and affordability has its place when selecting a test automation service provider, but the underlying value that is delivered should be equally, if not more important. When assessing value, it's always worth remembering:

When you accept less than you’re worth, that’s what people are going to keep offering you.

Luvvie Ajayi

Knowing what is available and what you need are two critical factors to understanding your worth. You are here because you want to be able to deduce what you really need from your test automation services provider. You have made a great start, now see it through to the end.

Does the provider use testing tools that suit your needs?

Not all software testing tools are created equal. If your organisation has already gone digital or is undergoing a digital transformation, then you need a fit-for-purpose automated software testing tool. In such situations, the right tool to help you achieve your goals is most likely not one of those that was popular a few years ago. What was that saying about an old dog and new tricks?

We always say that the best automated software testing tools include the following features, without exception:

All good software testing tools include test management, planning and execution. Only the best software testing tools also include automated test execution, automated scope selection and comprehensive debugging tools and analytics too.

Never accept a testing tool because it is presented as the "industry-standard" or most popular. Our guide to 10x Your Testing Effectiveness offers a list of verifiable factors that will help you understand what type of testing tool your organisation needs.

Before you complete this process, don't forget about load testing, especially for digital!

Want to see how our test automation tools use analytics to cut testing time to hours or minutes?

Schedule Demo Now

How will you objectively assess progress during the project?

Weekly meetings with carefully fudged curated PDF and Excel reports outlining project progress are tiresome, ineffective and achieve little. In a world where all relevant information should be at your fingertips, your automated software testing service provider must provide you real-time information about project progress. Information in "real-time" is key here.

The data points that you need visibility of during a project can be varied like those given below, but also should be able to be customised to your needs:

• Test script design and creation, including editable data sets
• Visibility of tests passed, failed and unexecuted
• Defect reports, particularly those that are overdue
• Verifiable results of test execution, including video logs, screenshots and technical results

Does the provider commit to measurable and achievable project outcomes?

Time and materials still has some place in today's dynamic digital landscape, but if you have laid your groundwork correctly, you should be able to work with your automated software testing service provider to agree on a fixed-fee and fixed-scope engagement. Such engagements compel both parties to work to reasonable timelines, concrete deliverables and measurable results.

You will likely have specific outcomes that you want from the project, but there should always be three key non-negotiable outcomes that your automated software testing as a service provider should deliver:

• Reduced testing time from days or weeks to hours.
• Increased depth and coverage of tests to find bugs before your users find them in production.
• An objective, data-based structure for verifying if an application is ready for release

How does the service provider deal with failure?

Things will go wrong in every project. New ideas will be experimented with and nuances of the application will need to be addressed. There is no way everything that your provider does for you will be a success. So will they cover up their failures or will they communicate with you about the best path forward? Will they be honest with you?

This is a very difficult characteristic to assess about a third party, but it may just be the most important of the six decision points on which you will make your decision. Churchill's line about failure has proven very instructive for when we have encountered it in our projects:

Success is not final, failure is not fatal: it is the courage to continue that counts.

Winston Churchill

One of our first customers from the United Kingdom is a very innovative company that placed a great deal of faith in our technology, but most importantly in our ideas. After a few weeks of work it became painfully apparent to our project leads that the path we had adopted was not going to provide any value to our customer. Our plan to tackle the project wasn't wrong, but our execution was lacking.

We (somewhat fearfully) communicated this to the customer and explained the reasons behind our conclusions. We not only explained what we would do differently going forward, but how this would affect the project's timelines. Most importantly, we delivered on this new plan! Four months after this change of direction, the customer was inviting us to help them with testing more of their application ecosystem.

Judging a potential service provider's mettle when the kitchen gets hot is a very difficult exercise. Get to the bottom of this before a project starts and you may just have a winning partnership.

Can the provider demonstrate an automation framework that will actually find bugs?

So you've got your testing tool, engagement model and end-goals sorted. Now, how will your selected solution provider ensure that the automated regression tests they build actually find bugs?

The single biggest factor that kills enthusiasm and future funding for test automation is a regression suite that merely cycles through a bunch of screens without actually testing for bugs. This often happens when you or your test automation solution provider relies on a "record and play" approach to test automation, with no regard for appropriate automated test design and model-based testing.

It is of vital importance to test the solution provider's technical expertise and their ability to marry technical skill with the core requirements of use journeys of your application ecosystem. Check out our guide to 10x Your Testing Effectiveness to understand the technical best-practices that your software test automation service provider should adhere to.

Can the provider back their claims with customer references?

This may seem obvious, but you will be surprised by how often reference checks are not performed. Even if you are 100% satisfied by what your provider of choice has demonstrated to you, it doesn't hurt to seek the views of people who have been in your position for some time.

The key topics to focus on during discussions with referees is around communications, timeliness, strengths and weaknesses in delivery. This information will help you forge deeper relationships with your provider in order to deliver better medium-to-long-term results.

Automated software testing is going to be an integral part of your future software delivery strategy and the decision to select a vendor to provide this service is key to your success. Download the Ultimate Guide to gain a detailed understanding of your key decision points. If you prefer talking to reading, speak to us about your goals and challenges:

Contact Us

Get Services Delivery Guide

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com