Background

A key tenant of modern software development is to engage your customers and end-users as early and as often as possible during the software development process. Experience has shown that by integrating your customers in the software development process, you have a higher likelihood of producing a quality product that meets the needs of the customer and industry.

The level of customer engagement will vary depending on the software and the target user community. In many cases, it is adequate to describe the software to users via PowerPoint presentations or the occasional web-delivered prototype demonstrations. Software can also be provided to end users in a "beta" pre-release format, though in many cases this happens too late to receive meaningful feedback from the users.

When an active and interested user community exists, many software development projects can be improved by providing the actual in-development software to a select group of users so that they can test the software with actual customer data, and provide essential feedback in areas of compatibility, speed, user flow, ease-of-use, and general value of the software to actual user needs. EPRI of course retains the ultimate decision-making authority and responsibility regarding the final product, but this feedback helps assure that the final product will be valuable to the industry.

This approach is implemented in the following process. This approach can be alternatively described as one element of "Agile software development or "rapid prototyping". In these approaches, rather than providing a "beta" release of the software when the software is nearly complete, the software is provided to a select user community on a frequent and recurring basis, possibly as often as every 2 to 4 weeks during the core implementation phases.

This process can be applied to both open source and closed source, licensed software development projects.

This process is distinct from the typical EPRI beta version process, which may still be used to provide the software to a wider set of users under a less restrictive licensing arrangement.

Challenges

Engaging external users in the development process is similar to requesting the review of a draft copy of any deliverable. However, providing early pre-release software to users comes with a number of risks and challenges that need to be carefully managed. Some of these are:

• The users selected to participate need to be sufficiently engaged in the software development activity so that they can provide insightful feedback within a short amount of time. This is not the place for users who just want to observe the current status of the development.
• Users need to be selected carefully, as those who do not understand the nature of the development might complain about the quality or direction of the software in public forums.
• The selected users need to represent the larger industry needs, and give a broad, holistic recommendation for possible features and avoid making recommendations based on self-interest.
• The selected users will need to be advised of the quality level of each iteration of the software. Even though the software may not be complete and not tested, users will expect the software to have a basic level of quality to maximize the use of their time and enable meaningful feedback.
• Since the software is not fully tested, the selected users need to understand that the software could damage their data files or systems, and take appropriate precautions.
• The users must not attempt to use the software for "production" purposes.

Specific legal challenges include:

• Participation in the software development and access to the pre-release software is not covered by the existing master license agreements that members have with EPRI, so additional legal language must be provided.
• EPRI staff needs to maintain our research role for the benefit of the public, and not develop software under this process for a single member, nor be unduly influenced by a single member that detracts from EPRI's research independence.

To address some of these challenges, the following process has been developed:

Process

1. The EPRI Project Manager, in consultation with the Program Manager, may determine that given the associated risks and costs, the project will benefit from a select group of users to participate in the development process. This determination may occur at the beginning of a project, or at some later time during the development of the project.
2. When the users have been identified, participation in this process, including the selection of development group participants, requires Director approval (or designee). The request for approval and authorization may be made by a simple email exchange. The approval is valid for a single development iteration (i.e. final release of a version of the software).
3. As of September 1, 2018, all EPRI pre-production (e.g., Beta) interim software releases must contain a pre-production acceptance splash screen. Users need to be given the option to accept (e.g., "Accept" button) or reject (e.g., "Reject" button) the acknowledgement. If accepted, the application loads; if they reject the acknowledgement, the application closes. The words "you," "your," or "your organization" herein each mean the EPRI-funding organization for which you are acting on behalf.
   

   This version of [SOFTWARE] ("Program") is a pre-release, interim version distributed in advance of EPRI's normal distribution for the purpose of obtaining user feedback and is subject to the terms and conditions of your agreement with EPRI, including confidentiality obligations use restrictions, and export control obligations. You may not distribute the Program to or use it for the benefit of any third parties. The Program (including information, data or results generated by the Program) is an "EPRI Material" and may not be used for any purpose other than providing feedback to EPRI. Your right to use the Program will terminate as stated in your agreement with EPRI or upon release by EPRI of a subsequent version of [SOFTWARE], whichever is sooner.
   
   

   The Program may contain errors or problems that impact your ability to use the Program, the accuracy of results generated by the Program, or may harm your system. You should promptly report any difficulties in using the Program, including, but not limited to, errors in the Program, assembly and run problems and/or inadequacy of documentation. EPRI may decide, in its sole discretion, whether to incorporate your feedback into subsequent releases of the Program.
   
   

   Please direct any questions or feedback regarding the Program to [PROJECT MANAGER] at [PHONE] or [EMAIL].




4. Prior to providing the initial interim release of the software to a development group member, the software must be checked by the EPRI SQA team. Subsequent interim releases will then be checked by the EPRI SQA team or other individuals designated by the EPRI SQA team. The scope of the review is:
5. • Does the software install / run on a computer other than the developer's computer (does not need to be a pristine environment)?
   • Is the information provided to the users with this interim release clear to explain what the current state of the software is?
6. The software will be provided to the member of the development group electronically via an authorized EPRI source that has access restricted to the participating development group members. This can be:
7. • FTP, uploaded by SQA
   • Secure E-mail directly from SQA
   • In the future, the ability to provide the software via the program cockpits or via a special developmental web portal may be provided.
   Note: The software must not be provided directly by a vendor to the member.
8. Members will be asked to uninstall/delete the software once the development period has ended (as required by the license). In the case of web-accessed software, the web access will be removed.