Questionnaire for Test Environment Management Requirements

Below is the list of questions that should be considered for managing non-production environments efficiently.

Test Environment Information

»        What is number of test environments and their purpose?  What is the configuration (hardware & software) of machines involved per test environment?  Are these physical machines or virtual machines?

»        Are there any dependencies between two or multiple test environments?

»        What were the main issues faced in past?  What are the pain points?

»        What is the operating system of machines used in test environments? Windows, Linux, Both etc.?

»        Are these test environments dependent on some third party environments or web service etc. (e.g. payment gateway, single-sign on)?

»        What is availability and utilization of test environments?

»        What is the network bandwidth usage? What is internet bandwidth usage?

»        How many machines being used for system under test and how many as test machines?

»        Do we have usage pattern of these test environments? Who has used it for how long? How many hours the machines were idle in a year?  What is the resource utilization?

»        Do you have any centralized inventory database?

Test Environment Management

»        How do you know which build is deployed at which test environments?

»        Which environment among all are updated most and why? What is the frequency of the update?

»        Which environment among all are updated least and why? What is the frequency of the update?

»        Is there any way of locking the test environment so that it can’t be tear down accidently?

»        Is there any process of shutting down the process if there are no activities in the environment for last x hours?

»        How do you know which environment is booked by whom and since how long?

»        How soon a new environment can be added in the existing cluster?

»        Can we restart and shutdown the environment automatically? If yes, is there any process of notifying for the outage?

»        What is the process of provisioning a test environment to some team?  How long it takes to provision the environment?  How long the environment can be provisioned? Is there any time limit?

Test Data Management

»        How test data is being managed? Is test data being shared across environment or a separate data for each environment?

»        How the test data being shared across environment? How PII data is being secured?  How test data being backed up and restore?

»        Are these environments having same database version (schema & data)? If not, what is the frequency of their updates?

»        How do you clean and refresh of test data?

»        What is your test data requirement?

Configuration Management

»        How do you make sure configured values of application is same and valid across all environment? How to identify any differences if there are any?

»        How do you know if the environment provisioned is having the right binaries and configuration?

Release Management

»        Do you have roll back plan for deployment?  Can you deploy previous version of application on demand easily?  How much time it takes? What is the process for doing the same?

»        Can test environment be selected and provisioned on demand? If not, how much time it takes usually to provision and deploy build to new environment?

»        How builds are getting deployed on test environments? What is the process of taking the changes from test environment to production environment?

»        How much time it takes to deploy builds in test environments? Is this an automated process or manual?  If it is automated, some commercial tool being used for automation or proprietary tool? If the process is manual, do you have any step-by-step instructions for the same? Is the build deployment process seamlessly integrated with continuous improvement process?

»        Is there any one click button for deploying the build to target environment? Like Octopus?

Accountability Management

»        Do we have audit in place? Are there any unauthorized changes to the environment?

»        How do you take care of security of test environment?

»        How do you install third party application in test environments? Is this centralized? Can a user install any utilities or software from internet or is required to take earlier permissions?  Are there dedicated resources for doing it?

Test Execution Management

»        Is this possible to execute test suite on selected environment?

»        Are test environments usually available during test execution? What percentage of time they were not available? How long team has to wait before it is available?

»        Can we schedule test at anytime from anywhere in the world?

»        Can same automation be executed against multiple existing environments with minimal changes?

Monitoring

»        How do you monitor the health of test environments and machines part of it? Is there any monitoring tool? Does it alert in case of test environment / machine being down?

»        Is there any dashboard suggesting, which environment being used by whom and for how long? Does the dashboard indicate active / passive test environments?

»        Are there any dashboard for monitoring the health of all environments and machines in it?

»        Are there any dashboard for monitoring software licenses installed and their version number? Does the dashboard suggest the usage history of this software?

»        How do you improve assets utilization by effective tracking and monitoring?

»        In case of test environment being down, someone gets notifications or not? If yes, what is the procedure of getting notifications and resolution?

Support Management

»        Is there any resource reservation dashboard?

»        What are your incident tracking mechanisms?

»        How do you administrator test users i.e. addition and deletion of test users?

»        Is there any single service window to fulfill all environment related requirements?

»        Who looks into the environmental problem and what is the process of raising the tickets?

»        How do we retire the hardware and renew it? What is the process? Do you have centralized inventory for the same?

Policies

»        What is the minimum time an environment can be shared and what is the maximum time?

»        What is the plan of data back process before decommissioning the environment?

»        What is the maximum duration for test environment usage? What is the average duration for the same?

»        Who will access the test environment?

»        Is there any SLA for provisioning the test environment?