DevOps is cutting the lines between production and software development. This modern approach to continuous delivery, integration, and deployment answers quick time-to-market needs. Testing in a DevOps world is often an afterthought. Those who tried to apply traditional testing methods in DevOps gradually failed because there was not enough time for test preparation and execution.
In this post, I will share some ideas about how testing could become a fundamental part of your next DevOps project.
Shift left -> testing for the expected
You've probably heard of "shift testing to the left." It's about early and continuous testing throughout the application lifecycle. Your testing teams review the requirements, document the test cases and the expected outputs, execute tests manually or electronically, and create test evidence reports.
A few things to consider
# Time for test execution is limited: Short-release sprints reduce the time to execute testing suites massively. We often have just one day or less to execute our regression sets. Risk-based testing and automation help to test the right things, even in limited time slots.
# Feedback loops: DevOps testing pays close attention to sense and response. You will only have time to review some test results in detail, create an excellent test report, and present your test summary after a few days. It is indispensable to be able to identify failed test cases and report those failures back to dev teams immediately.
# Flexibility: Quick adaptation of testing suites is critical. Projects in DevOps mode will deploy new releases frequently during testing stages. As a test manager or engineer, you have to find a way to deal with such permanent changes in your system under test. A flexible automation solution and framework can help you overcome such challenges.
Shift-right -> testing for the unexpected, undefined, unknown
In the case of "shift-right" testing, it moves towards production. Load Tests, reliability checks, user experience tests, and performance checks are executed during pre-production or production. A characteristic of all those tests is that the expected outcome is often unknown. Actual customer requests can drive your test cases, and you investigate how the new application behaves or simulates a massive production like the volume on your system under test. Focus your analysis on the undefined and unexpected, such as memory leaks, deadlocks, concurrency problems, exceptions, and misbalanced communication patterns.
A few things to consider
# Be creative: I can only give you a recipe that works for some of your applications. Shifting right requires an open mindset, a passion for data analysis, and creativity. Your success in this stage depends on how long it takes to identify hotspots and find solutions. Experience and the right tools can be a shortcut.
# Monitor all user requests: You won't have enough time for failure reproduction. Integrate APM and UEM solutions, which capture all user actions performed on your system under test, to avoid time-consuming error reproduction.# Reduce the risks: Don't release the new solution to your user community on day one. A technical go-live followed by a friend and family phase is often the preferred approach to balance the risk of reputational damage due to serious reliability issues.
# Review and improve: You built beautiful apps and ensured high reliability. Once the user and data volumes grow, response times can remain the same, and customers will move away due to the wrong user experience. Make regular performance reviews on production a fundamental part of your operational procedures to identify such slowdowns and initiate corrective measures.
New testing techniques
Learn from mistakes and improve your testing procedures daily. Below are some ideas for enhancing testing at the edge of DevOps. This approach goes beyond the traditional approach of testing for the expected, but it enriches classic testing.
# Real user requests
Time is limited in DevOps projects, and therefore, teams tend to deploy new versions early in production, even if not all functional test cases have been executed. Real users will use the new application differently with real data sets. Depending on the design of your new system, you can grant access to a limited user community and watch how those users work with your new application. Failures identified in this stage can give your developers valuable input and help them fix such problem spots before a broad user community gets affected. Furthermore, you should check if response times are within expectations and compare them with your non-functional requirements.
# Driving tests with production requests
Certain applications, such as message queue-based applications, often require heavy throughput. Adequate load tests are time-consuming, so creative minds store messages in production and replay them in their testing stages. The same testing with production requests can be achieved by duplicating user requests by using proxies.
# Vulnerability Scans
Quality is an important selling factor. Certain areas, such as dead code, complexity, or vulnerabilities, require static analysis. Extend your testing suite to vulnerability scans and static secure code analysis according to security standards such as OWASP's top 10.
Creative minds and forward-thinking leaders regularly release new testing utilities and practices. Schedule a slot in your weekly agenda for research and continuous optimization.
Keep doing the good things. Happy Testing!
Comments