Introduction

We are in an era where digital businesses understand the importance of building high-performance systems rather than reactive performance testing strategies. It is no more the core responsibility of the performance testing and engineering services team that works in silos to ensure the system performance is assessed and certified a few days before the move to production. With the radical shift to the Agile/DevOps development model, the adoption of early performance testing strategies (shift-left approach in performance testing) could bring exponential benefits to build and validate the system performance.

Whether it’s a legacy monolithic application, or an SOA (Service Oriented Architecture) application that consists of three server tiers – namely web, application, and DB servers, or a micro services architecture that consists of more fine-grained components to accomplish the same objective, a robust performance testing strategy is vital to identify bottlenecks at all components/layers. Distinct types of performance tests might need to be executed in the correct sequence to find performance bottlenecks in the system quickly.

What is performance test strategy?

A performance testing strategy is the first step of the performance testing process. It outlines in detail the objectives, scope, and approach for the performance test. The test strategy for performance testing also features the different type of tests needed to validate the performance and reliability of the application.

The test strategy should focus on end-to-end performance testing and bottleneck analysis covering all the layers – namely client-side performance testing (page loading and rendering time), network performance testing (impact due to network type), and server-side performance testing that includes load balanced distribution checks (equal load distribution analysis across the server farm), web server performance bottleneck analysis (HTTP requests and thread pool size analysis), application server/API server(s) performance bottleneck analysis (heap analysis and code profiling), and DB server performance bottleneck analysis (query performance analysis and profiling). This not only requires expertise in several testing tools and techniques to identify the bottlenecks but also expertise in providing performance optimization and capacity sizing recommendations (performance engineering).

A robust performance test strategy should focus on end-to-end performance testing of various layers to measure all the relevant Key Performance Indicators (KPIs) to certify a system for its performance and scalability characteristics.

In general, there are top #3 mistakes because production failures due to performance issues occur despite having focused performance testing activities:

• Quantitively and/or qualitatively invalid workload model (not derived from production usage patterns)
• Improper performance test environment sizing strategies (leading to improper extrapolation of test results)
• Use of low database volumes while performance testing (non-realistic data volumes used in DB)

A sound performance test strategy should consider these key aspects to make the right conclusion from the performance tests.

Prevalent performance bottlenecks at various tiers

Maintaining a database of frequently noticed performance bottlenecks of each tier can help performance engineers instantly connect the dots to drill down the bottlenecks. A robust test strategy for performance testing should also recommend running the right type of tests to quickly identify the bottlenecks at all layers of the system.

Here is a list of typical performance bottlenecks identified in various layers during performance testing:

Client-side (browser/mobile device) performance bottlenecks:

Client-side performance issues have become a key contributor to overall system performance during recent years due to the advancement of Rich Internet Applications (RIA). For mobile applications, analysis of device-side performance issues has become essential. Prevalent client-side(browser) performance issues are listed below:

• Complex DOM traversal and manipulations leading to high page loading time
• Large event handler bindings
• Slow CSS selectors and a high number of CSS/JS files
• Image handling issues
• JS code performance
• A high number of XHR calls
• A high number of network calls and requests served by a high number of domains
• High utilization of device resources – processor utilization, memory usage, battery usage

Network performance bottlenecks:

Network bottlenecks have become a major contributing factor particularly in mobile applications. Following are the major network performance issues that impact the performance of the application:

• NIC card configuration issues
• Impact due to Wi-Fi cellular networks – 2G/3G/4G
• Network bandwidth issues

Web server performance bottlenecks:

The web server serves the static contents (image files, CSS/JS files, etc.) to user requests. Prevalent web server performance issues are listed below:

• Server configuration issues
• Disk issues due to log size increase | Enabling of server logs
• Hardware saturation issues
• Limited throughput due to HTTP connection pool configuration issues
• Improper hardware capacity sizing to meet target user loads

Application server performance bottlenecks:

The application server handles the dynamic user requests and executes the business logic of the application. Prevalent application server performance issues are listed below:

• Ineffective application code
• Performance and scalability issues of third-party APIs
• Memory leaks
• Garbage collection issues
• Heap size configuration issues
• DB connection pool configuration issues
• Hardware saturation issues
• Application server configuration issues
• Improper hardware capacity sizing (& scaling strategy) to meet target user loads

Database server performance bottlenecks:

The DB tier has more performance bottlenecks when compared to other tiers. Prevalent database server performance issues are listed below:

• Cursor handling issues
• Ineffective SQL queries
• Issues due to cache settings
• DB server configuration issues
• Ineffective indexing
• Ineffective scanning
• Locks and deadlocks
• Hardware saturation issues

With the rise in new disruptive technologies and heavy competition across business players, there is an increase in the demand for enriching the end-user experience. Due to these reasons, there is an increase in the importance of proactive and continuous performance testing strategies to meet the ever-growing market needs. No doubt, the roles and responsibilities of performance test engineers are expanding, and they need to gear up to face the new challenges by adopting smarter, innovative, and continuous performance measurement techniques with appropriate shift-left and shift-right performance testing strategies that assure accurate performance assessment. This justifies why Performance Subject Matter Experts (SMEs) are required to strategize the continuous performance testing, engineering, and performance monitoring methodologies for modern enterprises.

Our Performance testing & engineering services (PT&E CoE (Center of Excellence)) at Nous-Testree has more than 20 years of industry experience in performance test solutions using various open-source and commercial performance testing tools. Supported by our performance testing and monitoring methodologies, client-side performance analysis strategies, differentiators solutions, Queuing Network (QN) theory-based accelerator tools, CI/CD enablement services, we have helped many of our clients in assuring the system performance.