Sample Performance Testing Run Report

                                               

Performance Test Report

For

XYZ Client

For 250 Concurrent Users

(1^st Run)

           

Revision History

A – Added, M – Modified, D – Deleted

S. No	Date	Version No	Page No	Change Mode (A/M/D)	Brief description of change
1	09-06-2010	1.0	All	A	First Performance Report – For 250 Concurrent Users
2	10-06-2010	1.1	All	M	Added observations
3
4
5
6
7

Table of Contents

1      Introduction.. 3

2      Definition and Acronyms. 3

3      Document Reference. 3

4      Test Environment.. 3

4.1       Load Generation Machines Configuration Details. 3

4.2       Test Data Details. 3

5      Description of the application.. 3

5.1       Technologies. 3

5.2       Functionalities. 3

6      Performance Test Goals and Requirements. 3

6.1       Business Transactions. 3

6.2       Performance Requirements. 3

7      Client Statistics. 3

7.1       Elapsed Time Vs Users. 3

7.2       Elapsed Time Vs Throughput. 3

7.3       Elapsed Time Vs Requests per Second. 3

7.4       Transaction Response Time Statistics. 3

7.5       Elapsed Time Vs Error Statistics. 3

7.5.1    Elapsed Time Vs Errors per Second. 3

7.5.2    Elapsed Time Vs Error Count. 3

8      Conclusions. 3

9      Recommendations. 3

10    Reference. 3

1         Introduction

ABC and XYZ worked together to evaluate the performance characteristics of mobile client application using REST protocol developed by XYZ. The main objective of the performance testing is to measure the client side performance metrics for checking the performance of the application at different loads.

The test was conducted at the ABC testing facility. ABC used JMeter for generating load using virtual users to simulate workload traffic against the servers.

The primary business goal of the application is that the application should sustain till 5000 concurrent users and at the same time finding it out opportunities to tune the mobile application.

2         Definition and Acronyms

Abbreviation	Description
Mbps	Mega bits per second
MB	Mega Byte
KBps	Kilobytes per second
TPS	Transactions Per Second

 

3         Document Reference

§         Performance Test Plan (XYZ_PerformanceTesting_TestPlan_2ndJune2011)

§         Proposal for Performance Testing Engagement to XYZ - ABC

§         Information provided by XYZ through several mails, teleconference discussions and documents

4         Test Environment

ABC used JMeter tool to generate the traffic on the server. Test runs were started with a small number of users. All users were added at predefined intervals. In the meantime, Client statistics were collected.

Test lab has been setups with master-slave configuration where there are around 20 slaves are being used for generating the load for 5000 concurrent users.

4.1     Load Generation Machines Configuration Details

Master & Slaves
Machine IP	Configuration
N/A	Operating System	Windows XP
	Machine specifications	Standard Desktop Edition
	RAM	2 GB

4.2     Test Data Details

The test data has been updated as per the suggestions provided by XYZ in this run.  The changes are highlighted in yellow below.

Test Data Detail	Facts
Duration of Performance Test Run (hh:mm)	01:30
Number of concurrent users	250
Think time for the performance run (secs)	3
Total number of users registered in data base	10000
Inbound Network bandwidth	4 Mbps
Outbound Network bandwidth	4 Mbps
Start time of the execution (GMT)	06/09/2011 07:32:00 AM

5         Description of the application

<< Short description of the application>>

5.1     Technologies

Architecture of the Application:

<>

5.2      Functionalities

The following scenarios were considered broadly during the performance testing of the application. 

Client Operation:

1.      Authenticate

2.      Sync Up

3.      Sync Down

4.      Logout

6         Performance Test Goals and Requirements

The goal of the performance testing project is to measure the performance of the mobile application and make sure the system is robust enough to sustain the load of 5000 concurrent users with around 83 requests / sec on server. In the current performance test run, 250 virtual users (used to generate the load) were run and the results were captured.

6.1     Business Transactions

S. No.	Transaction Name	ABC Scenario Steps
1	Edit Profile	Authentication à Edit Profile à Logout
2	Edit Agent	Authentication à FullSyncUp0  à Logout0 à Authentication1 à EditAgentSyncUp1  à Logout1
3	Adding Images	Authentication à SyncUp_AddingImgs  à FullSyncUp à SyncUp_DeleteImgs à Logout
4	Adding Images to Items	Authentication à SyncUp_AddingImagesRooms  à FullSyncUp à SyncUp_EditingRooms  à Logout
5	Adding Items	Authentication à AddingRoomsCategoriesSyncUp à FullSyncUp0 à AddingItemsSyncUp  à FullSyncUp1 à DeleteAllItemsSyncUp  à Logout

6.2     Performance Requirements

1. The applications must support 5000 concurrent users.
2. Average synchronization request should require less than 750 milliseconds of execution time on the server
3. When no data needs to synchronize, should take less than 500 milliseconds or preferably less than 250 milliseconds
4. The XYZ must be able to handle around 83.33 requests / sec

7         Client Statistics

7.1     Elapsed Time Vs Users

Observations:

• VUsers are being ramped up as expected i.e. total it takes 30 minutes to ramp up and then all 250 threads (VUsers) are running consistently.

7.2     Elapsed Time Vs Requests per Second

Observations:

• The number of requests / sec is not following VUsers ramp up and started decreasing after 20 minutes whereas it should also be increasing for 30 minutes and then should get stabilized.
• It can happen due to following reasons –
• There is problem in client side scripting and expected number of requests is not being invoked because of some client side failure.  But in that case, we should be able to see errors in the JMeter log file but we are not finding any.
• Threads (VUsers) are waiting for the complete response and subsequently sending next requests.  It seems to be the case in this case as response time has increased accordingly for steps (See Elapsed Time vs. Transaction Response Time steps).  Again the transaction response time here include network latency and server processing time.  The latency is due to network or server processing, we can get this detail only when we can analyze server side logs as well.

7.3     Elapsed Time Vs Throughput

Observations:

• The throughput is following requests / sec (Section 7.2), which is the correct behavior.
• It indicates that server is able to respond to all client requests the way it is coming and can handle more number of requests /sec.  In other words, the server is capable of handling 250 concurrent users and 25-30 requests / sec.

7.4     Transaction Response Time Statistics

Scenario 01: Edit Profile

Metrics	Min	Avg	Max	90%
EditProfile_Authentication	0.306	2.25642	14.192	5.289
EditProfile_EditProfile	0.317	2.510612	17.214	5.991
EditProfile_LogOut	0.295	1.767303	7.663	4.884

Observations:

• Response time of all steps have increased when throughput / requests per second has decreased.  As mentioned earlier, it might be possible that server processing time has increased when 250 concurrent users are in action or the delay is because of network latency.  It can be proved further by analyzing the IIS Logs file.
• There is high response time only for half an hour i.e. after elapsing 30 minutes to 1 hour.  After 1 hour, the response time came down drastically, which needs to be analyzed further.  Again it might be because of network or server.

Scenario 02: Edit Agent

Metrics	Min	Avg	Max	90%
EditAgent_Authentication0	0.31	2.243969	16.879	5.214
EditAgent_Authentication1	0.307	2.168309	18.135	5.123
EditAgent_EditAgentSyncUp1	0.662	5.092703	25.48	14.376
EditAgent_FullSyncUp0	0.802	14.52514	176.481	24.523
EditAgent_LogOut0	0.295	1.818143	13.379	5.021
EditAgent_LogOut1	0.294	1.820878	7.379	5.178

Observations:

• Same as scenario 01:Edit Profile (Section 7.2)
• Response time of full sync up getting deteriorated on elapsing time.  It needs to be analyzed further.

Scenario 03: Adding and Deleting Images

Metrics	Min	Avg	Max	90%
AddingImgs_Authentication	0.307	2.255168	15.004	5.293
AddingImgs_FullSyncUp	0.71	14.97312	403.949	25.435
AddingImgs_LogOut	0.293	1.804622	13.803	5.13
AddingImgs_SyncUp_AddingImgs	0.743	6.54148	60.045	22.111
AddingImgs_SyncUp_DeleteImgs	0.757	7.44002	49.482	24.63

Observations:

• Same as scenario 01:Edit Profile (Section 7.2)
• Response time of full sync up getting deteriorated on elapsing time.  It needs to be analyzed further.

Scenario 04: Adding Images to Items

Metrics	Min	Avg	Max	90%
AddingImg2Items_Authentication	0.306	2.221556	14.941	5.202
AddingImg2Items_FullSyncUp	0.732	15.57917	294.586	26.944
AddingImg2Items_LogOut	0.293	1.794356	13.796	5.051
AddingImg2Items_SyncUp_AddingImgRooms	0.768	7.109594	44.141	22.823
AddingImg2Items_SyncUp_EditRoom	0.746	7.474685	338.187	24.6

Observations:

• Same as scenario 01:Edit Profile (Section 7.2)
• Response time of full sync up getting deteriorated on elapsing time.  It needs to be analyzed further.

Scenario 05: Adding and Deleting Items

Metrics	Min	Avg	Max	90%
AddingItems_Authentication	0.307	2.216663	21.791	5.185
AddingItems_FullSyncUp0	0.66	15.4715	296.454	26.294
AddingItems_FullSyncUp1	0.679	15.0094	244.071	25.936
AddingItems_LogOut	0.294	1.869566	11.967	5.203
AddingItems_SyncUp_AddingItems	0.73	6.376414	34.423	19.814
AddingItems_SyncUp_AddingRoomsCategories	0.758	6.789037	42.688	19.358
AddingItems_SyncUp_DeleteAllItems	0.314	8.377027	55.582	21.836

Observations:

• Same as scenario 01:Edit Profile (Section 7.2)
• Response time of full sync up getting deteriorated on elapsing time.  It needs to be analyzed further.

7.5     Elapsed Time Vs Error Statistics

7.5.1       Elapsed Time Vs Errors per Second

Observations:

• The errors per second numbers are negligible (< 0.07 per second) and can be ignored for the performance run.

7.5.2       Elapsed Time Vs Http Status Codes

Observations:

• The http status code 200 & 204 (success status code) is following requests / sec graph (section 7.2) and which indicates server is processing all requests successfully.
• The http stats code 400 (bad requests) occurred 37 times during one and half hour load testing run.

7.5.3       Elapsed Time Vs Error Counts per Scenario

Scenario 01:  Edit Profile

ScenarioName_StepName	StepCount	ErrorCount	%Error
EditProfile_Authentication	949	0	0
EditProfile_EditProfile	948	0	0
EditProfile_LogOut	948	0	0

Observations:

• All steps in the scenarios have passed successfully, in other words the server have processed all requests successfully.

Scenario 02: Edit Agent

ScenarioName_StepName	StepCount	ErrorCount	%Error
EditAgent_Authentication0	948	0	0
EditAgent_FullSyncUp0	947	0	0
EditAgent_LogOut0	940	0	0
EditAgent_Authentication1	939	0	0
EditAgent_EditAgentSyncUp1	939	0	0
EditAgent_LogOut1	938	0	0

Observations:

• All steps in the scenarios have passed successfully, in other words the server have processed all requests successfully.

Scenario 03: Adding and Deleting Images

ScenarioName_StepName	StepCount	ErrorCount	%Error
AddingImgs_Authentication	5688	0	0
AddingImgs_SyncUp_AddingImgs	5685	1	0.01759
AddingImgs_FullSyncUp	5675	0	0
AddingImgs_SyncUp_DeleteImgs	5646	2	0.035423
AddingImgs_LogOut	5637	0	0

Observations:

• All steps in the scenarios have passed successfully except (AddingImgs_SyncUp_AddingImgs & AddingImgs_SyncUp_DeleteImgs), in other words the server have processed all requests successfully.
• AddingImgs_SyncUp_AddingImg and AddingImgs_SyncUp_DeleteImgs have failed less than 1% of time, which is very less number and can be ignored for the run.

Scenario 04: Adding Images to Items

ScenarioName_StepName	StepCount	ErrorCount	%Error
AddingImg2Items_Authentication	5676	0	0
AddingImg2Items_SyncUp_AddingImgRooms	5669	6	0.105839
AddingImg2Items_FullSyncUp	5660	0	0
AddingImg2Items_SyncUp_EditRoom	5621	11	0.195695
AddingImg2Items_LogOut	5612	0	0

Observations:

• All steps in the scenarios have passed successfully except (AddingImgs_SyncUp_AddingImgRooms & AddingImgs_SyncUp_EditRooms), in other words the server have processed all requests successfully.
• AddingImgs_SyncUp_AddingImgRooms and AddingImgs_SyncUp_EditRooms have failed less than 1% of time, which is very less number and can be ignored for the run.

Scenario 05: Adding and Deleting Items

ScenarioName_StepName	StepCount	ErrorCount	%Error
AddingItems_Authentication	5658	0	0
AddingItems_SyncUp_AddingRoomsCategories	5653	1	0.01769
AddingItems_FullSyncUp0	5639	0	0
AddingItems_SyncUp_AddingItems	5603	6	0.107085
AddingItems_FullSyncUp1	5588	0	0
AddingItems_SyncUp_DeleteAllItems	5554	10	0.18005
AddingItems_LogOut	5541	0	0

Observations:

• All steps in the scenarios have passed successfully except (AddingItems_SyncUp_AddingRoomsCategories, AddingItems_SyncUp_AddingItems & AddingItems_SyncUp_DeleteAllItems), in other words the server have processed all requests successfully.
• AddingItems_SyncUp_AddingRoomsCategories, AddingItems_SyncUp_AddingItems, and AddingItems_SyncUp_DeleteAllItems have failed less than 1% of time, which is very less number and can be ignored for the run.

8         Observations

• Requests per second are not consistent during complete run and have come down sharply whereas numbers of concurrent users were being ramped up.
• Throughput is following requests per second.
• Response time of all scenarios steps have increased when requests per second & throughput has decreased.
• Response time of ‘Full SyncUp’ getting deteriorated on elapsing time.
• Almost all requests are being served by the server successfully.  In other words, error percentage is very less (< 1%).

9         Conclusions

• The server is capable of serving 25-30 requests / sec with 250 concurrent users in terms of throughput and robustness (i.e. no errors).
• There are impacts on response time of all scenarios steps during the run in middle but cannot be analyzed completely without having server side metrics.

10   Recommendations

• Analyze IIS Log file of same window (load testing run window) for one and half and hour and rule out or find out any problems associated with server as the goal of this performance testing assignment is to find out bottlenecks present on server side rather than worrying about client side metrics (response time).

11   Reference

• The raw data that was used for preparing this report can be found at following ABC - FTP location with following details –

IP Address: 

User Name:

Password:  

Location: