Customer Acceptance and Independent Evaluation

The acceptance test, analysis, and assessment phase involves a structured handover of the finished software system to the customer organization. This ensures the customer's satisfaction and validates the system readiness for operational use. The development contract and project plan should specify what products are delivered, how they are documented, and how the customer will validate that the products are acceptable.

Without agreed-on acceptance criteria, a customer could hold off their formal acceptance, and usually their final payment to the developer, until an arbitrary amount of time has been spent doing extensive evaluation and possibly operational or production use of the system is performed.

Some development projects will specify that an independent verification and validation (IV&V) or an independent test team be part of the acceptance test process.

IV&V and independent test teams generally are examining, testing, analyzing, and assessing the system at the customer's direction, and this may be focused on some or all the systems requirements, on key elements of the design, or on critical sequences of planned operational uses of the system. An example of this would be to have a selected verification and validation (SV&V) team focused on security, reliability, safety, or other emerging properties or nonfunctional requirements of concern to the customer. This must be carefully managed.

Experience shows that the further independent teams are asked to move beyond the contractually binding requirements specifications in their analyses, the more likely it is that the issues they uncover will be genuine surprises to all parties. These will likely require compromise,

Acceptance criteria usually refer to the agreed-on requirements specification baseline. If that baseline correctly and completely captured security, safety, and reliability needs, then acceptance processes can confirm that the as-delivered system meets those requirements. These are usually validated by analysis conducted throughout the development process, which can and should use many of the techniques of a systems security assessment process.

contract changes, and additional time, effort, and costs-whether for development or to address business impacts incurred while working around the surprise factor-to remedy.

One form of customerdirected testing often used on larger, more complex systems is Operational Test and Evaluation (OT&E).

The independent OT&E team uses customerprovided operations concepts, plans, and procedures to develop and conduct an evaluation of the system in the same ways the customer-and their operations teamintend to use it. This is rarely used as part of formal acceptance testing but may be required prior to putting the system into fullscale production use.

Once all the acceptance processes specified in the project management plan have been successfully completed, including OT&E and security testing (if specified), the formal acceptance process is completed, contract acceptance documents are signed, and payments change hands as per contract terms.

The system is then turned over to its operators, who will take the system into production use.

Transition to Production

The end-user organization, rather than the developer, usually transitions the new system from the acceptance phase into the live production environment.

This transition may require obtaining security accreditation and providing training, awareness, and education to the new users according to the implementation and training requirements. Other activities would include implementing the system, installation and data conversions, and, if necessary, conducting any parallel operations.

Revisions and System Replacement

While systems may be in production mode, the hardware and software baselines should be subject to periodic evaluations and audits. In some instances, problems with the application may not be defects or flaws but possibly additional functions not currently developed in the application.

The end-user organization will hopefully have the system under formal configuration management, which will require that error correction, minor improvements, or the addition of new features should be subject to formal review and approval, including security review as appropriate. Whether this revision management process follows the same SDLC used during development or not is again something that the customer and end-user organizations must decide.

Periodic application audits should be conducted and include documenting security incidents when problems occur.