Testing performance requirements needs to be a fundamental task for every company. Now, it’s no longer a question of what type of application you’re developing or what budget you have. Plenty of tools are available—and thanks to the open-source initiative—we can now find some effective performance-testing products for free!
Interestingly, most decision-makers realize the huge benefits of performance testing only when a critical reliability issue hits their production environment.
It often takes a major disaster to convince those decision-makers to make performance testing part of the agenda for every project. Any change to mission-critical applications after that is validated for compliance to NFRs. Every time I witness this scenario, it never fails to surprise me how so many companies still neglect this crucial task. Let’s take a look at our traditional performance-testing use cases.
Use Case 1: Testing for compliance to performance requirements
Because in most cases, no performance requirements are in place, the performance engineers need to ask challenging questions about how the new system will be used and who will use it. The architects, business teams and developers will then outline the expectations for response times, throughput rates, the number of users and utilization. This is often the first time anyone in the project realizes how the application is not designed to manage the expected volume. At this point, the performance-testing teams start to compile a performance-testing strategy that describes which load patterns should be simulated, which test runs need executing and how performance testing will be embedded in the development life cycle.
Use Case 2: Testing for capacity validation
Operating applications at the wrong hardware sizing is expensive. Having insufficient hardware resources, is a waste of money; just like assigning a huge amount of CPU and memory. It’s always a good idea to use load and performance testing to calculate the system capacity required. At the end of the day, your savings will dwarf the amount you invest in such a testing exercise. Forward-thinking businesses have policies ensuring that every request for a new server has to include performance-test results to demonstrate that the requested capacity is in fact necessary.
Use Case 3: Testing for continuous improvement
Speed dominates every aspect of our lives these days. When a user waits for any longer than 3 seconds after selecting a URL, or after clicking a button on a website, they’ll simply move on to a similar service provided by a competitor. Any change in our application landscape can result in a negative user experience. That’s why continuous monitoring of key performance indicators is essential. Otherwise, we tend to spend the entire project budget in the first few months and ignore the real priority – monitoring, tuning, and improvements, once real users are using the application. All the savvy decision-makers have already closed this gap, with digital experience monitoring solutions in place to provide the insights required. Hotspots, error rates, stack traces, and user behavior are thereby visualized continuously, 24/7, on information radiators. If the automatically applied thresholds are exceeded, the site reliability team springs into action along with the performance-engineering team to run the necessary application tuning.
Creative engineers have recently come up with a few more practical scenarios to further widen the scope of load and performance testing. The focus is not limited to testing for response times or hardware sizing, but also allows you to select the testing approach.
Currently, our business teams or functional testers validate their features step-by-step. But is this the way the new application will be used in production? There are bound to be rare cases in which just a single user will use your new or changed application. Most of the time, though, hundreds or thousands of users will be working simultaneously. That means our testing does not align with the way an application is used post-production. Here are some ideas for new use cases of load and performance testing.
Architecture validation
We can be in for some big surprises when a new application is deployed to production if we rely too much on whiteboard architectures. The reason for this is two-fold. Firstly, application design is typically a one-time activity during the early development stages. It’s difficult to verify whether a developer has followed all the architecture decisions – any minor changes in the code can have a severe impact on call patterns or data loads. One of the best approaches for architecture validation is to make it part of your load and performance testing. Then you can simulate production volumes and check how your application stack behaves. Service-request volumes, utilization and transferred data volumes are some of the key figures you should collect before comparing them with the whiteboard-design decisions.
Web-analytics testing
Manually testing web-analytics features is not enough. In fact, the response times of your applications can have a significant impact on what your web analytics backend actually captures. Imagine your entire revenue forecast relies on web analytics, and due to a slowdown, the analytics integration captures only half of the requests. The decline in revenue that results will leave your C-level managers in a state of shock! Once they discover how the trouble was due to an analytics integration that wasn’t properly tested, they’ll hit the roof!. You can avoid such problems completely by integrating web-analytics testing into your performance-testing activities.
Identifying scalability limits
Outages of application servers, web, or database tiers are very hard to avoid these days. Response times can be affected if your application uses a database deployed on two servers and one of these servers crashes. If your testing approach doesn’t currently consider such scenarios, it’s time to close this gap. Then, in the worst-case scenarios, a subset of services and servers will still provide the required service to your customers. The resulting slowdowns are often acceptable in such situations, but outages should be avoided at all costs. Performance testing allows you to understand the critical path of your application. Don’t miss this opportunity in your performance-testing strategy!
Resilience testing
Applications may work well with only a small number of users. But once the applications have to deal with concurrent-user situations, they gradually fail. Such errors are difficult to catch and require a more advanced approach. This is especially true for financial transactions such as payments, which must be carefully tested for resilience. Once such transactions are safeguarded, the transferred money is refunded and the intended transaction, cancelled. Otherwise, the payment is initiated, but the purchased service is not delivered due to leaking software, and no refund is issued. You can avert such a disaster when you combine the failover scenarios with your performance-testing activities to make testing of such crucial scenarios a standard activity.
Production simulation
The first day in production should not bring any surprises to your operational teams. Make sure you give them enough time to get familiar with the behavior of the new applications. Understanding the critical path in such systems is key to the best possible user experience. With IT services becoming more complex these days, operators need to know which scaling approach has been implemented, as well as knowing the capacity limits within which the applications function well. Involve operational teams in your performance tests and let them see how and when applications break. This experience will help to make them more proactive in avoiding any bottlenecks for new services on production.
Validation of alerting and monitoring
One of the most important elements for reliable applications is very often overlooked. It’s the mechanism of how you collect the data that reveals what’s going on in your application and which actions you should take, based on any changes in your monitoring insights. It’s important to strike the right balance when setting up alerts. Go for the minimum. Otherwise, if your monitoring system is very noisy, you’ll risk the alerts being ignored altogether. Performance testing gives you the chance to test your monitoring and alerting systems under real-world conditions.
Why not share with me any other great benefits you know about from conducting load and performance testing?
Happy performance engineering!
Kommentarer