Get up to speed fast on the techniques behind successful enterprise application development, QA testing and software delivery from leading practitioners.
Shift your UI testing to sprint zero by leveraging mockups and AI
Fast fixes for slow tests: How to unclog your CI pipeline
8+ programming languages that will keep you in demand
Here’s the right way to test links in your web apps
Microservices and COBOL: 4 steps to modernize your apps for DevOps
Trends and best practices for provisioning, deploying, monitoring and managing enterprise IT systems. Understand challenges and best practices for ITOM, hybrid IT, ITSM and more.
Why most machine learning projects stumble
Digital transformation fails: Are you automating the wrong processes?
Why ITSM is key to your digital transformation
3 container orchestration challenges to tackle head on
5 principles for your cloud-oriented open-source strategy
All things security for software engineering, DevOps, and IT Ops teams. Stay out front on application security, information security and data security.
Put more ‘Sec’ in your DevSecOps
Rethink your DevSecOps: 4 key lessons for achieving security as code
Economics meets cybersecurity: A light at the end of the tunnel?
BSIMM12 is here: 9 key takeaways for software security teams
The shift from DevOps and security to DevSecOps: 5 key roadblocks
TechBeacon Guides are collections of stories on topics relevant to technology practitioners.
TechBeacon Guide: DevSecOps and Security as Code
TechBeacon Guide: SecOps Tooling
TechBeacon Guide: World Quality Report 2021-22
TechBeacon Guide: The State of SecOps 2021
TechBeacon Guide: Application Security Testing
Discover and register for the best 2021 tech conferences and webinars for app dev & testing, DevOps, enterprise IT and security.
Webinar: Maximizing Your IT Assets
Webinar: Get a Fast Pass to Full-Stack AIOps
Webinar: Access Mainframes Securely from the Cloud
Webinar: Best Practices to Protect Data in the Cloud
Webinar: Threat Hunting—Stories from the Trenches
You may be familiar with one of these software testing scenarios.
After you have watched the tests get slower week after week for a year, suddenly someone realizes that testing is constraining delivery and demands that something must be done.
Congratulations. You are something.
Or maybe you have just started a test tooling adventure, the tests are running quickly, and you would like to keep it that way.
Either way, we thought you could use a little help.
TechBeacon asked nine experts in software testing for their advice on how to speed up the testing process, focusing on how to speed up a continuous integration (CI) pipeline bogged down by tests. Here’s what they had to say.
Dave McNulla, staff software engineer, Teradata:
I hate waiting for test results! If tests run too slowly, I look to run in parallel (or expand parallel execution), which requires the tests to be completely independent. Adding more systems, whether cloud or on-premises hardware, costs money, but so does making developers wait for results. That involves labor costs, opportunity costs, and the cost of delay to the business. It is a triple threat.
To speed up tests, I often look for “sleeps” in test-ware. They waste time, and they are indicative of a flaky test. The test author does not really know what to wait for and ends up waiting for an extended period of time. If the test is important, there are ways to poll more often so the test resumes sooner.
After that, I look for low-value tests to run less often. Value is calculated by likelihood of failure paired with the priority of a bug found by a failure. If a test never fails or only finds bugs of low priority (or redundant with another test), we can run it less often, perhaps overnight or weekly.
Tim Ottinger, senior technical consultant, Industrial Logic:
One solution to slow tests is to trade integration for speed. If the system under test calls for a database query, replace the query with a mock that returns a prefab dataset instead. This saves the network and database time and should be good enough as long as your dataset meets the criteria you expect from your database query. Try moving slow tests down Cohn’s test pyramid. If you are doing a system test, could it be an integration test? Could the integration test be replaced by a few good microtests? The lower-level the test, the faster it typically is.
Also, don’t use Gherkin only to drive the GUI. We find that Gherkin-based tests work great as an alternative user interface to an application. It will call the same functions the real GUI would call yet is much faster.
Also, realize that if your tests require you to stand up a full application with all its supporting services, then you’re going to be subject to long startup/shutdown times. If you can, avoid that sort of full environment, or try to do it just once for the full test run, not on every test case.
The faster your tests are, the more often they’ll be run, and that’s a key performance metric for developers. The tests they don’t run do not catch any issues, and the less often they run them, the more confusing it is to understand what is wrong. Fast tests are run frequently, and this helps developers work in smaller, safer steps.
Finally, consider in-memory databases and small datasets. This one simple trick has paid significant dividends.
Don Jackson, chief technologist, application delivery management, Micro Focus:
Some people state that the solution for speeding up testing is to shift it to the left, toward development of the application. Others say it is shifting to the right. But what needs to occur is shifting together. Organizations need to automate their testing and incorporate it into their CI/CD pipelines. That automation needs to have contributions from all three personas that have a vested interest in test automation: the traditional QA automation engineer, the domain expert/business analyst (shift right), and the developer (shift left).
When constructing the tests, organizations need to focus on the business-critical process flows/paths to mitigate the business risk and ensure that checkpoints are not run more than once for the same object that is in the same state. Furthermore, the pipeline needs to leverage the concepts of distributed testing, which allows the suite of tests to be executed on multiple nodes, concurrently. Lastly, exploratory testing should occur during the execution of the pipeline by the team, adding the human/unpredictable element into the testing and finding those oddball defects prior to release.
Marisa Shumway, vice president of product marketing, CloudBees:
Feature flags can improve your testing process in several ways. For example, QA can use the flags to turn off behavior to compare it to previous releases. Without this, getting a developer to change the code and create a new build could cause a delay of hours or days, and, realistically, the test just might never run. With the flag override view, QA can shorten the feedback cycle by having complete freedom to toggle flags in the environment with the click of a button. Now QA is empowered to work independently, without asking developers to break their focus or waste precious business cycles. Once the flag exists, if there is a problem, a rollback is a simple configuration change. This eliminates the need for retest/rebuild, making the rollback process take seconds instead of hours. That reduces the risk exposure, which allows the team to proceed with less delay.
Philip Ross, senior software engineer, Synopsys Software Integrity Group:
Once you’ve identified slow tests, review each, and if there’s no obvious bottleneck, run the test with a profiler to see the breakdown of where the time is being spent. With that information, hopefully you can see any hot spots and can then work on fixing them.
Consider checking your configuration and seeing if there are developer-like performance settings in the test environment. Some of these developer-like settings may have traded runtime performance for compile-time performance or something else, which you may not want on a static CI test run.
Also, check to see if there are hard-coded constants to set per environment. For instance, a sleep for retrying HTTP requests should be set to 0 in a test environment where you’re not hitting a real endpoint; you can fix that by moving the sleep amount into your configuration and setting it to 0 in the test environment.
Michael Larsen, senior automation engineer and Socialtext show producer, The Testing Show:
Have a look at what your tests are actually doing. A good rule of thumb is that any test should do exactly one thing unless otherwise warranted. This atomic nature is the key to creating unit tests, and it’s just as important for other integration or end-to-end style tests. It’s common that slow tests are overloaded—they try to do too many things. By paring down the steps each test has to perform, the overall time for each test goes down, and often there can be a synergistic effect. You may find that running multiple single-purpose tests is quicker than running one overloaded test.
Andrew Knight, lead software engineer in test, PrecisionLender:
One of the best ways to speed up tests is to run them in parallel. Attempting to shave one second off this test here and three seconds off that test there doesn’t have the same impact as running two tests concurrently instead of serially. Most test frameworks already have built-in parallel capabilities. Enabling parallel execution is frequently just a setting or a command-line option. However, the real challenge with parallel testing is avoiding collisions. If tests share any resources, such as users, services, or databases, then there is a chance that they could access shared resources at the same time, thereby “colliding.” For example, in a store app, if one test tries to retrieve an order right after another test deletes it, then the first test will fail.
The best way to avoid collisions is to make sure tests do not modify any shared data. If a test needs to change data for testing purposes, then it should create new data for it to use exclusively. Going back to the store app example, tests should always create new orders instead of reusing existing ones.
Lalitkumar Bhamare, senior software test engineer, XING; CEO, co-founder, and editor, Tea-time with Testers magazine
I see a lot of organizations creating more problems with CI than they solve. In my experience, CI-related things that drag organizations behind are poorly implemented CI jobs and not giving enough thought to what amount of automation should be done at what layer. In addition, poor infrastructure can lead to flakiness, while ever-failing automated checks that block deployment create an ongoing mystery to solve. That can lead to engineering time and productivity wasted on “fixing” the CI pipeline, plus firefighting caused by environment/infrastructure problems that do not reflect actual production failures.
If that sounds like you, one quick fix I can offer is building CI jobs so they can only run selective automated checks—for example, only for the areas affected by a particular code change. That will speed up the test run and prevent false failures, and it is cheap. Another option is to stop kicking off builds for nonproduction code changes, such as a change to “helper” scripts or changes in the automated checks.
James Bach, consulting software tester, Satisfice:
The first thing I do is to center myself. I remind myself that I might, in fact, be a bottleneck. I then remind myself that being a bottleneck is not necessarily a bad thing. (Security is a bottleneck in airports. Is anyone saying, “Let’s get rid of security because I want to save 15 minutes”?). I reflect on how I may not be good enough at my job, however, and that the perception that my work is too slow could be valid.
Once I am centered, I want to understand the perception. What, specifically, seems slow? To whom does it seem slow? Slow compared to what?
This is a process of entering the system that I need to deal with.
As I learn what the concern is all about, I may find that I agree. But a common problem is that people judging the speed of testing haven’t thought through the value of the process. Testing is like insurance, in the sense that you don’t get insurance because you hope to profit. You get insurance as a hedge against loss. Testing has a cost in return for that investment and time—we gain some probability of discovering important and evasive problems. We do our best to match the investment to the potential risk.
When it comes to the test process, there are the things that need to run all the time (the CI pipeline), the usual process of testing, and the occasional processes, such as a security audit, that can slow everything down when they are kicked out.
Our experts’ advice boils down to a few things: Remove things from the pipeline, speed up the pipeline, improve the general process, or reduce the impact of the special/odd things, such as security and performance testing. It might be possible to bring those into the continuous integration pipeline or to enable developers to do that locally before the pipeline runs.
In any event, if you want to go faster, you can lighten your load or build up your muscles. If you routinely use sleep statements or use high-level tests, congratulations! You might have some easy, fast fixes.
Everyone else may have to do some flexing and heavy lifting.
Special thanks to Lee Hawkins, whose suggestions to run in parallel, use mocking, test at the right level, remove waits, and do one thing well reinforced the ideas of the contributors.
Take a deep dive into the state of quality with TechBeacon’s Guide. Plus: Download the free World Quality Report 2021-22.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon’s Buyer’s Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon’s Guide.
Get the best of TechBeacon, from App Dev & Testing to Security, delivered weekly.
Brought to you by
I’d like to receive emails from TechBeacon and Micro Focus to stay up-to-date on products, services, education, research, news, events, and promotions.
Check your email for the latest from TechBeacon.