Identification of Performance Objectives / Goals

It is found that most of the time, stakeholders are not aware or clear on what they want after the performance run. After performance run, every performance test engineer prepare the report containing many graphs, analysis and what not and share it to stakeholder. But stakeholder is confused or overwhelmed by viewing the big report. The main reason is he or she is not sure what answer they were expecting in the report and it becomes very difficult for performance tester to explain or walkthrough the report. Therefore, it is very important that before we conduct performance run(s), we set the expectation in the beginning that what question(s) will be answered after the run. Below are some of the questions, I ask before conducting a performance run and prepare test strategy and report accordingly.

Client Side Goals

• Do you want to know if the application is able to support X number of concurrent users load?
• How yours most important transactions and pages performing in terms of response time during normal load and peak load?
• Would you like to know if X number of transactions can be completed in Y number of minutes?
• Is new version of the software adversely affecting response time (create performance baseline)?
• Among all key transactions and pages/screens, which are the top 5 slowest transactions and pages that can be considered for optimization for next release?
• Do you want to know if response time of all important transactions and pages are less than X sec?
• Do you want to know size of web page / screen or the number of objects on the page impacting performance for end users? If yes, then which objects are the affecting it most and what are the alternatives?
• Do you want to know If yours most important transactions and pages are reliable (error rates less than 5%) during normal load and peak load?
• Are you worried that yours most important transactions and pages / screens are not available to customers during peak load time?
• If performance (response time and error percentage) of yours most important transactions and pages / screens might degrades after application is up and running for a longer duration (say 8 hrs?)
• Would you like to know after application get stressed out it comes to normal state by itself when no requests are coming?
• At what point does the performance degradation occur?
• Would you like to know how many concurrent users and transactions per second will crash the system?
• Would you like to know the performance behavior of application during spike (all concurrent users get ramp up in very less time)?

Server Side Goals

• Do you want to measure resource utilization like memory, cpu, disk i/o and network i/o of servers (web server, application server and database server) during normal and peak load?
• Would you like to know if application server has memory leak?
• Would you like to know what parts of the system perform poorly and under what conditions?
• Would you like to know which configuration (Garbage collection, Connection Pool, Thread pool etc.) provides the best performance?
• Would you like to find out the recommended configuration for iis server and database servers for peak load traffic?
• Would you like to know how many transactions / sec (throughput) being handled by different servers during the peak load?
• What kinds of errors are generated on application server during peak load?
• Would you like to know where the threads are waiting in application server when response is very slow?
• Would you like to know what is being processed when performance of the application is found to be slow?
• Would you like to know what code snippet is slowest during peak load on application server?
• Would you like to know top X slowest SQL queries during the peak load?
• Would you like to know the root cause of performance degradation?
• Would you like to know the performance of most important transactions and pages at various database size?
• Would you like to know which layer is performing slow?
• Do you want to know when there is a time to put better hardware or additional hardware to satisfy the future user base and its load?
• Would you like to determine if application is capable of handling future growth?

Technology or Stakeholder Specific Goals

• Would you like to stress every web method separately and create a baseline?
• Would you like to conduct stress testing of database stored procedures?
• Would you like to know how cache is being utilized on different servers?
• Would you like to know the performance behavior of application during various connections behavior (persistent, retrying etc.)?
• Would you like to know the performance behavior of application when all users abandon the site without logging out?
• Would you like to know performance behavior of application when accessed from different locations?
• Would you like to know performance behavior of application when accessed from different network bandwidth (home users, office users, etc.)?
• Which types of users (home, administrator, business etc.) are affected most during peak load?
• Would you like to know the performance (response time) of page for usable components only or full page load?
• Would you like to know which request (among thousands) has taken maximum time and its breakdown?
• Do you want to know are there any third party providers such as rating, reviews, ads, news feeds, e-commerce engines and CDN hurting the performance?
• Would you like to reproduce some performance issues in the system? What would be that?
• Do you have some specific performance issue that you want to solve? What would be that?