Performance Testing Elicitation Questionnaire

For a vendor offering performance testing service, submitting a competitive proposal and winning pursuits is not an easy task.  Submitting a proposal usually requires approach, estimate, cost and assumptions for the engagement and it’s a challenge for service vendor to get this information correct in less time with minimal information available at hand.  It is seen that sometimes vendor ask their prospective customers lots of questions, which might not be relevant for preparing the proposal but might be good for the overall performance testing engagement.  In this article, I will discuss minimal set of questions that should be asked and importance of these questions for preparing the proposal.  These questions are not the only questions and based on the answers, further relevant questions can be asked.

Project Context / Testing Objectives

Question	Notes
Please provide business overview of Application under test	It helps in understanding the application’s domain and its associated challenges / risks. » High volume (image, videos) transfer in Healthcare application » High number of transactions in BFSI applications » High session length for e-Learning application
Do you have any performance testing concerns?	Not every time it is required to do load testing for end-2-end application.  Sometimes the goal of performance testing could be very specific - » Validation of failover scenario while there is a load on the system » Identification of best configuration for the application server setup » Reproduction of production (performance) issue in test environment » Server getting choked after a while
Do we need to analyze and identify bottlenecks in the system?	Not every performance testing goal is to identify the bottlenecks, sometime creating a baseline is enough.  In case of bottleneck analysis is required, monitoring tool and selection of monitors become important.  The cost of engagement can be more due to monitoring / diagnostic tool in use and effort required for analyzing and identifying the bottleneck.
What types of performance testing are expected? (e.g. Load Test, Stress Test, Failure Test, Endurance Test)	Helps in designing approach for execution pattern and identifying number of performance runs and its duration.  The number of performance runs and its duration help in calculating the effort correctly. High level description for different testing types is given below. Load Testing – It is conducted for validating the performance characteristics of the system when subjected to workload / load volume what is anticipated during the production load. Stress Testing – It is conducted to make sure that application can sustain more load than anticipated on production. Failure Testing – It is conducted to find out the capability of the application in terms of load. The load (concurrent users or volume) is increased until the application crashes. Soak Testing - It is conducted to find if system is capable of handling expected load without any deterioration of response time / throughput when run for a longer time.
When would you like to start the engagement? and when would you like to finish it? Please provide key milestone details	It helps in finding number of business days left for completing the engagement. If release date is near and effort required is more than resource and schedule need to be planned accordingly.
Did you conduct any performance testing for the application under test in the past? If yes – can we have the performance report for the same?	Any past performance testing experience information is useful while preparing the approach of the engagement.  » Any gap from past performance testing can be filled » If stakeholders are comfortable with the past report template, same can be used » Can be learnt from past mistakes
Is the application available over the internet? Can you provide us access to the system for conducting the PoC? If not, can you provide us a walk through of the application for better understanding?	It helps us to understand complexity of application under test better and submitting the proposal with confidence

Load Profile / Transaction Information

Question	Notes
Do you have list of key critical scenarios that requires scripting? If not, would you like us to identify critical scenarios during the engagement?	Number of critical scenarios help in calculating effort of scripts designing correctly.  When identification of critical scenarios is also in scope, effort of analyzing log files and interviewing stakeholders increases accordingly.
What is the average and peak concurrent users? In case concurrent users information not known, please provide below information for calculating the same. Visit Rate (visits / hour) Visit Length (minutes / visit)             OR Page Rate (pages / hour) Number of Pages in a Scenario Scenario Execution Duration	It is one of the most important information for generating required load to application under test and licensing cost associated with virtual users
How many transactions per second?	It helps in validating the number of concurrent users requirement.  It might be possible that goal is to simulate number of transactions and not the concurrency, in that case virtual users licenses cost can be optimized.  In some cases, mentioned number of concurrent users won’t be sufficient for simulating number of transactions and will demand further clarifications from customer.
What is the network throughput (bytes/sec) between client and server?  In case throughput information not known, please provide below information for calculating the same Approx. Number of Scenarios Avg. Number of Steps (Pages) in a Scenario Avg. Number of Bytes per Page	It helps in calculating bandwidth cost for the engagement.  Consider an application where thousands of files (images, videos, 100 page reports etc.) are being uploaded and downloaded.

Technology

Question	Notes
What servers and network topology being used for application under test?  Can you provide us the system / physical / architecture diagram of application under test?	For large scale distributed system, large number of different Operating Systems (Windows, Linux, Unix etc.), different platforms (IIS, Tomcat, Websphere etc.) and database servers (Oracle, DB2, MySQL, SQL Server etc.) are used.  It is very important to understand this architecture for preparing the approach for the engagement.  Analyzing large number of servers requires effort and cost increases accordingly.  It is best to get system / physical / architecture diagram to understand the same.
What are different technologies at client end? (e.g. AJAX, Simple Web, Flash, ActiveX, Silverlight etc.)	Selection of load generation tool is dependent on this information. If it is a simple web application than open source tool like JMeter can be used but if it is a SAP controls than costly tool like HP Load runner is needed.
What are different technologies at middle tier (e.g. NET, IIS, Tomcat, Web sphere)?	Selection of monitoring tool and respective monitor is dependent on this information.  If the technology is new and not many monitoring tools exist than effort required for diagnosing the technology would be additional.
What are different technologies at backend (e.g. Oracle, SQL Server, DB2 etc.)?	Selection of monitoring tool and respective monitors is dependent on this.
What are different operating systems being used for servers? (e.g. SQL Server on Windows, Tomcat on Linux etc.)	Selection of monitoring tool and respective monitor is dependent on this information.
Protocol used for communication between client and server? (e.g. HTTP, SOAP, REST, Citrix, RDP, VPN etc.)	Helps in identifying the load generation tool. The application might be simple web based application but accessible only over the VPN, in that case approach for performance testing will differ.
Are there any other components or 3rd party applications that require special coordination? Please describe.	Approach of performance testing will differ if - » CDN is being used for serving the static contents » System is integrated with third party gateways like payment gateway » Instead of physical servers, virtual servers are being used » Apart from application under test, there are other services running » There are other batch jobs running on database server

Test Environment / Infrastructure Information

Question	Notes
We are assuming that there is a dedicated test environment available for this performance testing engagement.  Please confirm.  If test environment doesn’t exist, do you expect vendor to setup the test environment. If yes, please provide information on number of servers, hardware configurations, software, software editions and version.	Test environment is the prerequisite for conducting the performance runs.  In most probability the test environment is provided by customers but in some cases it might be possible that customer is looking for setting up the application under test by vendor.  Of course, it increases the cost for hardware and software licenses and effort for setting up the lab.
What is the maximum number of (transactional) records in database before starting the performance run? Do you expect vendor to prepare Test Data baseline?	Populating production like data in test environment is a huge effort.  Consider populating million of transactional records in database.  Usually, it is done through backup and restore but in some cases due to security restriction, this is not possible and test environment database requires base lining with other approaches.
We are assuming, required number of physical or virtual machines will be provided to us for load generation environment setup.  Please confirm.	This question become important when number of load generation machines required is more (say >5).  If load generation setup is on Customer network, it is better to make them aware about this cost.  If load generator machines are on vendor premise or on cloud than additional cost and effort will be incurred for setting up the load generation environment.
We are assuming vendor team will be provided access to test environment.  Please confirm.	Sometime due to customers company policy, it might not be feasible to provide access to application under test environment and if bottleneck analysis is in scope than there are lot of activities like monitoring agent setup will lie with customer team.  The customer should be aware about this and if all their labs are being managed by third party vendors than it might incur additional effort due to communication overhead.
Do you have any existing load generation tool (e.g. HP Load runner, JMeter, Neoload)?  How many virtual users does it support? What protocols does it support? Does it have seat license or floating license?  What is your preference for the load generation tool (open source or commercial tool)?	License owned by customer is a big plus for preparing the proposal as it reduces the cost significantly.  Existence of license can also mean that scripting might be already existing and there is no need of conducting tool feasibility. Though, not every time, it might be correct, especially if maintenance of script is in scope as usually it is seen that designing script from scratch is faster compared to correcting the existing scripts, which were not executed for a longer time. In absence of load generation tool, it is important to know customer’s preference than suggesting your own tool.  It might be possible, customer is only comfortable with HP Load runner and not with JMeter even when the application in scope is simple web based.
Are there any existing monitoring or diagnostic tool (e.g, HP Diagnostic, Dynatrace, AppDynamics, New Relic)?	Diagnostic tool becomes important when bottleneck analysis is in scope. For large scale distributed system it is preferred to go with commercial tool, which incur additional cost. If customer doesn’t have any diagnostic tool nor interested in paying for the same, rather would like to use free monitoring tools that come along with operating system / platform (e.g. perfmon, sqlprofiler, vmstat, VisualVM etc.), diagnostic efforts will increase accordingly.

Communication Process / Support Information

Question	Notes
What time of day can the site be load tested? Will there be any additional users accessing the application during the test assignment?	If weekends or late nights are required than cost and effort may vary.  If other users are also using the application, additional communication and analysis overhead would be incurred.
Do we require any special permission prior to generating load of a certain volume?	If it requires a big process to engage everyone before starting the performance run, additional effort will be required as communication overhead.
Who administrates your current performance-testing environment of the application under test? load-generation environment?	If it is being administered by third party vendor than additional effort will be required as communication overhead. If it is being managed by customer team, SLA should be maintained. If it needs to be managed by vendor team addition effort will be required.
We are assuming that customer will provide access to the server systems to monitor server-side performance counters. Please confirm.	Sometime due to security concerns, vendor team will not be provided access to live monitoring and they will be dependent on offline data for analysis.  It should be assumed that customer team will share the correct information after the execution.  Effort may vary due to this communication overhead.
If database inserts are generated during the testing process, Please confirm that your support team owns the process of refreshing the data accordingly	If it is the responsibility of vendor team than effort will increase.
We are assuming that the test data for the transactions will be provided by customers.  Please confirm.	While simulating the scenario, it is required to use varying input data (e.g. login, credit card details, invoice number etc.).  Sometime, number of these records goes in thousands and there are data dependency among these records.  It is a time taking process, if these records need to prepared by vendor by creating new transactions and hence would incur additional effort for this task.