Applications that perform
Almost no-one needs selling on the importance of performing applications. While functionality and features are important, if they’re not performant they will either be abandoned altogether or accepted grudgingly by an increasingly sophisticated user-base. Maybe even worse, they may cause issues in the rest of the application or system.
Modern applications are often made up of a hybrid of technologies, interfaces, cloud-providers, OS, networks. If your mouse-click ends up with a spinning wheel, do you have any idea where the problem occurred? Poor network, API bottleneck, local browser, cloud-hosting service?
To anticipate performance issues in production, Application owners and Test managers generally conduct performance or load tests in an environment as close as possible to production in order to root out any bottlenecks or surprises before go-live of a new version.
The challenges of a good load test
Test Managers face two major challenges when trying to test for performance.
Firstly, when they are designing the tests they need to understand the production load profile i.e. how many users access the application at the same time; when are the peak loads: what sort of transactions are they conducting; how long do they stay; what’s the current response time; what’s the API-load; how is the current production system handling this load?
Secondly, when they do find poor performance during a test, they want to be able to deep dive and understand the root cause of these performance issues: is it a bottleneck with network, threads, database, poor code etc.
Test Managers regularly turn to Performance and Load Testing tools to help them manage their performance tests but often miss the benefits that could be provided by an APM Tool (Application Performance Management tool).
APM Tools – a brief look
While performance testing tools are usually in the domain of the Test Manager, APM tools may be governed by the Operations organization. In the past these may not have been immediately available to the application development team, but with establishment of DevOps best practices as well as the reduction in tool cost, the access to and associated benefits of APM tools are making them easier to use within the delivery lifecycle.
The full scope of APM tools differs from tool to tool, but in general an APM tool collects and monitors information about what’s happening in the application. The principle components of any APM solution usually include the following:
Performance Monitoring: Performance of individual web requests or transactions. The test manager can use this to understand current transaction performance and to ensure that there is no system degradation during testing.
System Usage: Usage and performance of all application dependencies like databases, web services, caching, etc. The test manager can use this to help get a picture of how the application is currently used. This information can then be fed into the design of the load test.
Transaction trace and code performance profiling: These tools are essential for assisting a developer to trouble-shoot performance bottlenecks.
System and Application metrics: APMs provide a host of data about server metrics such as CPU, memory as well as application framework metrics like performance counters, JMX mBeans, etc. For continuous delivery situations, test managers can use this information to understand the current production footprint. During testing they can evaluate how the new application release compares to the benchmark.
APM tools have many other features which lie outside of scope of testing – availability monitoring, application architecture discovery, alerts/event correlation, real-user monitoring. More and more, APM vendors are delivering AIOps tools that aim to get insights into operational incidents, and ultimately to predict failures before they occur.
Don’t forget the essentials
It’s easy to get bogged down in the details of a load test with all the issues related to data set up, synthetic users, scripting issues, system statistics, test reports. The best test managers always come back to the essentials, which is being able to answer the question of how the system will perform come go-live and what will the user experience be. Good load and performance testing certainly goes a long way to answering these questions.
However, getting the feedback loop on actual production performance as input to a test, coupled with the insights provided by APM tools during a test give the Test Manager the confidence to certify the upcoming release.