Chapter 2: Analysis, Design, Development, Implementation, and Evaluation

This page is a section of TC 7-102 Operational Environment and Army Learning.

There are five phases the training and curriculum developer progresses through to develop and institute training, education, and leader or self-development material. The analysis, design, development, implementation, and evaluation (ADDIE) process takes the training and curriculum developer from the initial determination of a requirement or need through the analysis, design, and development phases; to implementation of a learning experience; and continues a learning continuum with evaluation improvements of training or education products-process. The required outcome of this continuum is a sustained and/or improved readiness to standards.

Introduction

The analysis, design, development, implementation, and evaluation (ADDIE) construct is a generic process with which to conduct instructional system design. Adult learning principles are central to creating and sustaining an effective continuum of learning. The decision to integrate operational environment (OE) conditions in the development of new training and education products, or during the revision of existing training and education products, bridges an analysis of task or action to be experienced with the standard to be achieved. The Operational Environment Enterprise (OEE) is central to supporting the training and curriculum developer with OE conditions and resources. (Figure 2-1 summarizes the ADDIE process.)

Figure 2-1. ADDIE training-education-leader development continuum

As training and curriculum developers integrate information and resources to best support identified requirements, the value of the OEE is most evident in the analysis, design, and development phases of the ADDIE process. Nonetheless, OEE considerations affect all five phases of the ADDIE process through the formative evaluations of each phase and the continuum of summative evaluation. The OEE is a repository and provider of multiple resources that support credible conditions required for quality training, education, and leader development.

Examples of OEE products and services are located in  this chapter and chapter 3 “Operational Environment Enterprise (OEE) Support” of this training circular. Integration and updates to the OEE products and services are a recurring factor of training and curriculum plans and programs. The training developer and curriculum developer recognize that OEE support can be focused in elements of the Army Learning Model (ALM) and on the people involved in the learning experience. Requirements have an impact on the development and proficiency of the trainer and educator, and the preliminary knowledge and/or skill sets required of the learner. The ways and means for both of these groups is amplified by how the training or education is designed, developed, and administered. Whether training or education, the environment of learning occurs within live, virtual, constructive, and/or gaming domains. (See figure 2-2.)

Figure 2-2. Training and education implication for Army readiness

Threat Managers provide threat and OE support to training and curriculum development. As subject matter experts, their support assists in the determination on what level of OE complexity will be required for a task/action and standard. The Threat Manager or a designated OE subject matter expert (SME) is involved at each level of the ADDIE process to ensure an accurate and appropriate OE condition to a particular learning requirement.

Identifying resource requirements are critical points during the entire ADDIE process. Early analysis and planning for resources are key factors in order to manage the time involved in securing the necessary materiel and/or expertise. Proponents identify resource requirements, availability, and constraints for equipment, facilities, funds, personnel, and time. Resource analysis incorporates learning strategies and implementation of the Army Learning Model (ALM) 2015. Allocation and commitment of resources support a level of fidelity required of a particular learning episode or event. Three levels of fidelity (high, medium, and low) in collective training also have applicability to other learning environments. These resourcing levels are:

• High fidelity is the condition-setting training environment of capabilities and resources needed to replicate most complexities of an OE, present realistic signatures and effects to stimulate all combined arms decisive actions and unified action enablers, and produce ill-structured problems for leader development. These tasks are set within the context of achieving all multi-echelon unit training tasks and objectives.
• Medium fidelity is a reduced condition-setting training environment of capabilities and resources needed to replicate the majority of OE complexities to stimulate key combined arms decisive actions and unified action enablers, and present partial signatures and effects needed to stimulate primary multi-echelon tasks and training objectives.
• Low fidelity is the minimal requirements and resources needed to replicate OE conditions that drive single echelon collective training tasks and objectives.

The ADDIE process improves a learner-centric model and institutes the most effective learning methods for the required outcomes based on the analysis of the audience. Learning environments vary dependent on the training or education need and available resources. These environments include self- structured, peer-based, informal social-based network, immersive, and/or formal academic or training venues. An example of condition is a cyber environment and requirements for training and/or educational experience and learning. In order to exemplify the integration of OE conditions into training and education development, this chapter builds a vignette based on a gap in cyber operations (see figure 2-3), and traces the development of particular conditions through each phase of the ADDIE process. Fundamental learning model considerations in these examples and/or similar experiences are―

• Mission command and unity of effort in the conduct of U.S. Army mission command philosophy.
• Collaborative and/or individual knowledge-gathering and/or problem-solving experiences.
• Tailored learning episodes and events programmed in a  logical sequence  for experience and expertise development.
• Conduct of mission tasks and orders with a clear understanding and application of commander’s intent, prudent risk-taking, and disciplined initiative.
• Blended technology-enabled instructional approaches that combine variations of live, virtual, constructive, and gaming (LVCG) in learning appropriate to a particular experiential environment.

Figure 2-3. Cyber gap introduction (example)

Core competencies for Army Soldiers, leaders and civilians are criteria to assess and evaluate particular learning experiences. Although comprehensive evaluation of competencies may not be appropriate in every learning experience, one or more competency is integral to any Army learning experience. Learning outcomes incorporate competencies and attributes in a learning experience and can include―

• Character and accountability.
• Comprehensive fitness.
• Adaptability and initiative.
• Attitude of lifelong learning.
• Teamwork, team-building, and collaboration.
• Communication and engagement.
• Critical thinking and decision-making.
• Creative inquisitiveness with intuition-innovation.
• Cultural awareness and understanding.
• Tactical proficiency.
• Technical expertise.

Sections 1 through 5 of this chapter provide a discussion on functions in each phase of the ADDIE process, and present considerations on how to integrate conditions of a selected OE in training and/or education. Considerations of “how to” for the training and curriculum developer and commander or leader are reinforced with sample questions and aspects at Appendix A, “OE Integration Job Aid for the Army Learning Model,” and Appendix B, “Exercise Design Checklist.” Concise vignettes throughout chapter 2 and chapter 3 illustrate examples of how a developer can use the five-phase ADDIE process to integrate operational environment conditions with resources and support learning with the Operational Environment Enterprise (OEE). An illustration of each ADDIE phase spotlights ideas of how OEE resources can support the progressive training and/or education programs for Army readiness.

Section I - Analysis

Analysis is the ADDIE phase used to analyze and determine a need for training and/or education. The primary analysis processes used to identify the learning products to be designed (revised or created), developed, implemented, and evaluated are needs analysis, mission analysis that includes target audience analysis and job analysis, and outcomes analysis. Analysis provides information about─

• Skill or knowledge requirement to be trained or learned, and either sustained or improved.
• Conditions in which the requirement is to be performed or the knowledge used.
• Standards of performance to be achieved and maintained.

Analysis is, in general, a process of investigating a triggering circumstance that indicates a state that is less than a required or expected capability, or suggests an actual or perceived performance shortcoming or deficiency. When a gap in acceptable performance or operation is identified, analysis determines how that change might or does affect critical Army operations and how the problem area can be solved. The training and curriculum developer conceptualize the impact of the changes on how Army Professionals ethically, effectively, and efficiently conduct unified land operations. The curriculum or training developer must integrate the principles of the Army Ethic in the process of identifying the required skills, knowledge and corresponding tasks. When required, complementary actions translate the impact of a gap into required skills and knowledge requirements, and corresponding tasks or actions that remedy Soldier and unit skills or knowledge proficiency to an acceptable standard of performance. Analysis may identify problem sources other than training and education such as policies, motivation, or inadequate resources or materiel. Any one of these could drive a requirement to revise training or education venues.

Proponents focus on essential and critical requirements and desired outcomes. Developers specify a target audience and compile a prospective list of individual or collective critical tasks. The developer integrates lessons learned from Army experiences, experimentation, and testing venues. Proponents sustain a continuum of needs analysis and ensure horizontal and vertical alignment of lessons learned as well as emergent information and observations into training and education products. (See figure 2-4.)

Figure 2-4. Analysis phase of ADDIE and OEE considerations

The analysis phase begins with a triggering circumstance. The triggering circumstance may come from a variety of sources in the form of a problem to be resolved. Analysis of the circumstance can result in identification of a training or education gap. However, not all triggering circumstances result in a need to change training and education. They may change other DOTMLPF factors. While there are innumerable potential triggering circumstances, generally triggering circumstances will fall into one of the following categories:

• Change in OE conditions. The triggering circumstance here may be the result of changes in current conditions in the security environment, anticipated changes in a future OE, emerging capabilities of potential adversaries, or anticipated advances in technology available to U.S. or potential adversaries. While it may be intuitive to think that this area is where OE considerations are most impactful, that is not necessarily true. The triggering circumstance only begins analysis. Regardless of the trigger, the training and curriculum developer must determine the training or education gap in terms of tasks, conditions, or standards. It is likely that the result of an analysis in a change of OE conditions will result in a gap in learning conditions, but it could also result in tasks or standards gaps.
• Change in DOTMLPF factors. The DOTMLPF domains are used to determine the capabilities required for building and employing military forces. They are extensive and changes will require the training and curriculum developer determine potential impacts on training and education programs. Changes to the way the Army operates (doctrine), changes in force structure (organization), development of new training requirements, new equipment fielding (materiel), development of new leader attributes (leadership and education), changes in manning or staffing levels (personnel), or training area closure (facilities) can result in identified gaps. Any one of these changes, with the possible exception of facilities, has to be placed in the context of an expected OE and the interaction of that environment (including threats, terrain and weather, populations, etc.) with Army forces in order to understand the potential impact on training and education.
• Lessons learned from operations. During the analysis phase it is important for the developer to consider and incorporate approved lessons learned. For the Army to be a learning organization that develops Soldiers, leaders, and civilians for operational adaptability, the institution must be able to rapidly integrate lessons learned into training and education programs. The lessons learned will usually take the form of new Solider or unit tactics, techniques, and procedures (TTPs). In most cases, the OE is a significant component of any lesson learned often manifested as a change in OE conditions.
• Inadequate results of current training or education based on evaluations. This category is essentially the T of DOTMLPF. When a training and education event produces poor results, institutional evaluations or reports from operational units must analyze and identify the areas that must be corrected. Analysis reviews the basics of task, conditions, and standards. A primary OE consideration is to determine if the conditions for the training or education event are appropriate, complete, or realistic enough. When significant change occurs, there may be a periodic requirement for course revision. A course may become obsolete or outdated for many reasons, including changes in the security environment that invalidates current TTPs. Lesson revisions are mandatory immediately when task performance threatens survivability, mission accomplishment, or a major environmental or safety impact is identified.
• Introduction of new methods of instruction or instructional technologies. Training and education is impacted by new technologies, cognitive science developments, and evolving instructional techniques. These changes can develop gradually or appear rapidly in this learning domain. Correspondingly, the complexity of the OE and the Army need for Soldiers, leaders, and civilians who can apply judgment and operate effectively within this complexity, requires realistic training and education conditions. This convergence may drive training developers to blend fundamental learning techniques with technology-supported methods to improve instruction and the learning that results.
• Changes in plans and policies. Senior leader decisions can change the requirements for training and education even absent the factors above. These changes could take the form of mandated tasks, restrictions in time, resources, or budget, or change in focus among others. While a change in the OE may drive new plans or policies, it is important to establish the specific conditions required to meet the identified training or education gap. (See figure 2-5.)

Figure 2-5. Cyber gap vignette-Analysis (example)

The resources available from the OEE have a direct impact on analysis outcomes among collective and individual training and related professional education. Nonetheless, several fundamental aspects of learning model analysis are universal. Analysis components of an ADDIE process include:

• Goal analysis identifies how the course supports the institutional mission, vision, and learning outcomes. Analysis results in a course goal or goals.
• Topic analysis specifies topics that support identified goals and learning outcomes.
• Target audience analysis describes students' current knowledge and experiences, and suggests effective training/education delivery methods.
• Gap analysis compares the desired learning outcome of the above topic analysis with the student's pre-instruction knowledge determined in a target audience analysis.
• Resource analysis identifies resources, availability, and constraints.
• Preliminary evaluation and milestone planning, and a tracking system of the developmental processes may include OEE elements; however, the gap and resource analyses are critical to identifying what is available to set the appropriate learning conditions.

Needs Analysis

Needs analysis determines what is needed to solve or mitigate known or anticipated gaps between current Army capabilities and required Army capabilities. The training and/or curriculum developer focus on comprehending the foundational knowledge and critical tasks or actions required in the learning. A learning outcome states the level of ability and type of competence a learner will achieve at the conclusion of a learning experience.

Analysis begins with an understanding and appreciation of current and projected future operational environments. Resources include results of current training or education, based on evaluations that indicate inadequate performance to standards. New technologies, cognitive science developments, and evolving instructional techniques and other learning skills can rapidly change a learning experience. Conditions such as threats knowledge, capabilities and limitations, and technology advances or limitations on the ability of people to perform actions or tasks shape the outcomes requirement. These conditions may indicate a requirement for new methods of instruction or instructional technologies.

Resource limitations and/or constraints are a norm. Learning can be affected by constraints such as maximum allowed course length, manpower availability, budget allocation, and required student load. Other typical areas for resource analysis include access to training areas, classroom information technology capabilities, training simulations and simulator limitations, and available time. Senior leader decisions can change the requirements and availability of resources for learning experiences. These changes could take the form of mandated tasks, restrictions in time, resources, or budget, or change in learning focus.

Mission Analysis

Mission analysis studies and determines the specified and implied tasks or actions that must be performed to standards in order to accomplish a mission. Mission analysis is a continuous analytic process centered on a mission essential task list (METL), unit task list, job or topic requirements, or other specified directives. Factors that prompt mission analysis include but are not limited to a─

• Revised operational concept and employment doctrine.
• Mission change of a unit, organization, or activity.
• Known, emergent, or anticipated threat in an operational environment.
• New weapon system and/or other military materiel.
• Degradation or improvement in personnel and organizational skill sets.
• Different operational environment factors (PMESII-PT) that effect a mission.
• Changes in the DA Standardized METL for the parent higher echelon organization.

The mission analysis output is the Unit Task List (UTL). The UTL is a list of existing collective tasks or emergent collective tasks to be designed and developed for a specific unit-type based on missions identified in the table of organization and equipment (TOE). The UTL reflects terminology for company-size and higher echelon units as stated in FM 7-15, Army Universal Task List (AUTL) with changes. See TRADOC Administrative Publications web site for specific “how to” information related to mission analysis. Mission analysis data is an output that may also determine the need for additional job analyses, individual tasks, and/or collective tasks and actions.

Outcomes Analysis

Outcomes analysis is required for all training and course design or course revision. Proponents identify the expected outcomes that signify successful completion of an action or task. Outcomes must describe what an individual, unit, organization, and/or activity must be able to be, know, or do within a set of operational conditions.

Command involvement is integral to the development of training and education outcomes. Outcomes incorporate the commander's or leader's intent and clearly state the focus for training and education and level of observable demonstration and/or performance to standards. Leader development is an overarching perspective to required outcomes. Command approval completes each phase of outcome analysis until the next formal review of outcome analysis. These outcomes must include the Army Profession General Learning Outcomes and align with the Army Ethic in a way which produces Soldiers, Army Civilians, and Leaders capable of making credible decisions during peace, war, and persistent conflict.

Section II - Design

During the design phase of the ADDIE process, data analysis is transformed into a working template for training and education products. The design phase uses the outcomes from the analysis phase to scope the parameters of design. The developer identifies the objective(s), which vary according to the type of instruction to implement, in order to satisfactorily address a requirement or gap. The developer confirms learning objectives with the approval authority, plans what training/instruction will look like when it is complete, and describes the context in which the task or learning will successfully occur. (See figure 2-6.)

Figure 2-6. Design phase of ADDIE and OEE considerations

Design identifies the task or action to be learned, the conditional situation for a particular task or action, and the required measures of performance to achieve a standard. The developer determines learner assessment methods, lesson sequence, methods of instruction, and media and/or other criteria needed for learning. Design produces the details of when, where, and how outcomes are to be achieved. Tasks and actions are usually categorized in one of three learning domains. Those three learning domains are cognitive, affective, and psychomotor.

The developer takes the OE perspective determined in the analysis phase, and selects the best methods of available resources to achieve necessary conditions in order to accomplish essential tasks and critical tasks to desired outcomes. Design identifies relevant OE considerations and requirements such as―

• Role players that accurately portray noncombatants, threat regular forces, and threat irregular forces for training and education.
• Training aids, devices, simulators, and simulations (TADSS).
• Terrain.

Figure 2-7. Cyber gap vignette-Design (example)

The training and curriculum developer uses the products created during the analysis phase to identify the course and lesson components. During this phase, the developer translates topic lists into terminal learning objectives (TLOs) and supporting enabling learning objectives (ELOs). Design effort arranges TLOs and ELOs into a progressive and sequential learning order. Initial assessment plans describe how an individual or collective unit/activity achieves standards to the prescribed learning level. (See figure 2-7.)

Individual Design Requirements

Before the design or redesign of a course, the training and curriculum developer identify prerequisite student knowledge, skills, and attitudes to ensure that the training and education outcomes meet the needs of the target audience. Design must consider the target audience profile developed during analysis and tasks/topics from previous training or educational experiences. A critical aspect during this phase is consideration of the impact on changes in the strategic environment that affect tasks or actions.

Training and curriculum developers specify any unique faculty/instructor requirements for knowledge, skills, and behaviors required to teach a particular learning experience. In the design phase, tasks and/or topic lists from the analysis phase are translated into learning objectives. Learning objectives provide details used to determine mastery of content. The learning objective―

• States the learning contract among the students-learner, faculty, trainer, and/or responsible and accountable learning organization.

• Serves as the foundation for educational design.
• Provides the basis for instructional training method.
• Determines instructional content.
• Frames learning conditions.
• Identifies standards for measurement or assessment of learning.

There are two types of learning objectives in course or lesson design. The two types are terminal learning objective and enabling learning objective.

Terminal learning objectives (TLOs) are the major tasks and topics identified during analysis that are quantified in a task or action, condition, standard format. The TLO is the main objective of a learning experience and is the performance required of the individual, unit, or activity to demonstrate competency in the task or action. The TLO describes exactly what the student must be capable of performing under the stated OE conditions to the prescribed standard(s) on lesson completion. There is only one TLO per lesson regardless of the presentation method or media, and the description has one verb. The TLO may cover one critical task such as a skill or knowledge, or more than one critical task (CT). (See table 2-1 for elements and considerations in defining a learning objective.)

Table 2-1. Learning objective requirements

Learning Objectives
Learning Objective Elements. All learning objectives must include a task or action, condition, standard, level of learning, and domain.
Task/Action. An action statement specifies what a student should do as a result of the education. Use only one verb in any action statement. The verb selected for the action statement must be congruent with the level of complexity of the action described.
Condition. A condition statement describes the learning environment that includes what will be provided (a scenario, small group, etc.), what will be withheld (without references, closed-book, etc.), any time constraints, and issues that affect student learning. (For example, issues can be material, equipment, special conditions, references, the role the individual, unit, or activity.)
Standard. The standard statement provides the criteria used to measure how well an individual, unit, or activity masters the learning objective. It minimizes subjectivity in measuring student attainment of the identified skills. The standard should incorporate minor topics identified in the topic analysis.
Learning Level. Base the level of learning on the complexity of learning. The desired level of  learning dictates the selection of the verb in the action statement and is reflected in the criteria used to assess student learning. Many educational institutions use Bloom's Taxonomy when selecting the cognitive level of learning. In addition, when the objective indicates assessment of the affective domain, the level must be the same level as the cognitive domain.
Domain. Three learning domains are cognitive, affective, and psychomotor. Educational institutions generally use cognitive and affective domains. Training situations can apply all three learning domains.

Enabling learning objectives (ELOs) are the fundamental subordinate tasks or actions required to achieve the TLO. ELOs are quantified in a task or action, condition, standard format. They are the supporting learning objectives identified in the task/topic analysis. ELOs must be learned or accomplished to standard in order to achieve the TLO. At least two ELOs support a TLO. Each TLO must be unique and is not used as an ELO. The training and curriculum developer format all learning objectives according to the requirements that state a task whereas an educational curriculum objective states an action.

Collective Design Requirements

Collective training prepares cohesive teams and units to accomplish their missions in the range of military operations and maintains a high state of readiness throughout the train/ready and available portions of the ARFORGEN cycle. It is delivered in units and institutions. Collective training requires interaction among organizations or individuals to perform tasks, actions, and activities that contribute to achieving mission-essential task proficiency. Collective training includes performing collective, individual, and topics (leader tasks) associated with each training objective, action, or activity. Use collective training design considerations listed on the Army Training Network (ATN).

Design collective training concurrently when determining system and TADSS training strategies and available capabilities. Events that support collective training enhance individual, leader, and unit or activity task performance proficiency and teamwork. These events may be conducted either in units or during resident training. Collective training events are the integral element of a combined arms training strategy (CATS) that prepares Soldiers, leaders, and teams to accomplish their missions in a known or contingent decisive action environment.

Resource and Milestone Planning

Validate resource requirements identified initially during analysis and verify requirements during each stage of the ADDIE process. Adjust for changes in requirements, availability, and constraints for equipment, facilities, funds, personnel, and time.

Resources are further identified when designing or revising the learning steps and/or activities. Training and curriculum developers notify responsible officials of resource shortfalls identified during the process and state the impact on mission accomplishment. As priorities of support are determined and used to refine design of learning outcomes, total training resource requirements or estimates eventually reside in the appropriate individual training plan (ITP), course administrative data (CAD), or program of instruction (POI) in the approved automated development system. The ADDIE process is a recurring way to maintain and adjust learning experiences and outcome expectations based on the resourcing decisions of responsible officials. Other actions include regular update of milestone plans for scheduling, additional requirements, or potential issues that affect a learning outcome.

---

Note. When the nature of a training experience allows the training-exercise planner to develop an opposing force (OPFOR) order of battle (OB), task organization, and equipment tier levels for a credible and robust threat in training, use of the HQDA TC 7-100.2 and TC 7-100.3 are ready- resources of unclassified threat data and tactics. For composites of actual weapon systems and other equipment capabilities, scoped for a training environment, see the Army Training Network website under “CTID Operational Environment Page” and “OPFOR & Hybrid Threat Doctrine.”

---

Section III - Development

Development is the production phase of ADDIE. The training and education developer validates resources, and confirms if availability of a resource has changed. Given a change in resources, the training developer reenters the design phase to reconstruct the lesson or course plan to achieve desired outcomes.

Developers take approved design outputs and turn them into completed, approved, validated products including details required to implement the instruction, assess the students, and evaluate the program. Major aspects in development are to―

• Develop  the lesson plan and  advance sheet  for educational  venues.  A  training environment expresses learning outcomes in a training plan or forms such as an operations plan or order.
• Develop and refine the specific individual, unit, or activity performance or ability to confirm that the learning outcome is achieved.
• Develop  an  assessment plan that  measures successful  achievement  of  the task/action  to  the standard.
• Review and update required and available resources with which to conduct the learning experience.
• Revise the milestone plan based on current assessment of resources and training and/or education preparations. (See figure 2-8.)

Figure 2-8. Cyber gap vignette-Development (example)

The trainer and/or instructor prepare to implement the approved products. The proponent command authority approves the final plan, lesson, course or learning experience for implementation. A development strategy includes identifying all materials that support the implementation of a learning product. Developers verify that available resources satisfy the conditions required for the skills proficiency in the selected OE conditions.

The Threat Manager or OE subject matter expert assists in the developer review and validation process for credible and robust operational environment conditions. Figure 2-9 describes the major activities of the development phase of the ADDIE process. OE consideration on resources may include but are not limited to:

Figure 2-9. Development phase of ADDIE and OEE considerations

• Common Framework of Scenarios (CFoS).
• Operational Environment Assessments (OEAs).
• Decisive Action Training Environment (DATE)
• Regional case studies, handbooks, and/or topic vignettes.
• Training circulars with training and educational information.
• Observations and lessons learned from live training and operational missions.
• Constructive, virtual, and gaming simulations.

Conditions, Circumstances, and Influences

Verifying that all required resources are available is critical in this phase to ensure the required conditions are in place for the expected learning experience and outcome. The conditions affect how the task or action can be presented with the appropriate impact of situational circumstances and influences.

The term operational environment can be misunderstood as applicable to only the operational level of conflict, and that the operational variables used to describe an OE are applicable only at the operational or joint level. Both of those views are incorrect. The correct understanding of operational and variables means that an OE and its variables can relate to an operation at any level of tactical, operational, and strategic mission.

The training and curriculum developer determines the combination of variables to focus a particular learning event. Selected subvariables further define a learning environment and provide considerations for how conditions can be applied to a task or action. The integration can range from a student presentation on the operational variables of a current event to a practical exercise in a selected vignette to culminate the learning activity. Table 2-2 is an example from TC 7-101 of the political variable and several subvariables to illustrate options available to integrate into a particular learning experience.

The eight operational variables and their associated subvariables describe the OE in terms that relate to specific situations as well as to threat capabilities. The variables are relevant to every echelon of command and every military mission within decisive action.

While individual variables do not dominate every environment, the variables are all present and require careful consideration and application to present the required credible and robust conditions. The operational variables produce a coherent profile of the conditions that can be applied to educational experiences, individual and collective training events, and small-unit or large-scale multilevel training exercises.

The training and curriculum developer analyze and verify all resource support required for the learning experience. Upon completion of this phase, all the conditions for the training and/or education are ready to be implemented. A discussion of the various OE products and services support available from the OE Enterprise is in chapter 3, “Operational Environment Enterprise (OEE) Support.”

Table 2-2. Political subvariable setting considerations (example)

Definition: Determines the type of government structure and associated behaviors encountered.
Subvariable Setting	Setting Definition	Considerations and Additional Notes
Dictatorship/Authoritarian	A mode of government characterized by the existence of a single ruler or group who arrogate to themselves and monopolize power in the state, exercising it without restraint.	For live training requires robust domestic security apparatus, bureaucratic institutions and bloated role-player government structure (1.5 normal manning).
Theocracy	A government ruled by or subject to religious authority. A system of government in which God or a deity is held to be the civil ruler.	Can only occur in the first two categories of social religious diversity.
Representative Government	Characterized as a representative form of government—either democratic, republic or parliamentary in form—with elected representatives and executives. All politics are governed by will of people and government has limited, defined powers over the population.	Can be democratic, republic or parliamentary. Requires specific, prominent influencers, council members be present in the scenario (in addition to the normal role-player government positions).
Anarchy	Absence of any form of political authority. Political disorder and confusion. Absence of any cohesive principle, such as a common standard or purpose.	Tribal and religious role-players present in scenario. Small percentage of disenfranchised or former government officials with little influence.
Subvariable Links: Economic:  Illegal  Economic  Activity;  Social:  Education  Level,  Religious  Diversity,  Criminal Activity.

Resource Analysis Considerations

PMESII-PT is a memory aid for the eight operational variables that make up any OE. When listing the variables, it is therefore advisable to list them in the same order as in the memory aid. However, this sequence does not mean that one always addresses and analyzes the variables in a particular order. The order in which variables are considered may depend on several things, including whether or not the—

• Training or educational experience will use live, virtual, constructive, and or gaming enablers.
• Operational theme(s) selected for the training or educational experience requires specific conditions in one or more variables.
• Training and/or curriculum developer is analyzing an actual OE, modifying an actual OE, or creating a composite OE for training.
• The  training  unit, institution,  or  activity  requests  specific  conditions in  which to  perform tasks/actions.

A key decision in developing the learning experience is to determine the type or types of training enablers to create the required fidelity in conditions. This decision may affect the order in which variables are considered, assessed, selected, and applied. For example, if the learning activity is to be conducted in a live training venue, many aspects of the physical environment variable are already set. When virtual, constructive, or gaming enablers are involved, the physical environment variable might be considered only after settings for other variables have been determined.

The type of training enablers to be used to reinforce the individual learning activity is critical during the development phase. Table 2-3 [at next page] provides a sample overview of the types of enablers and learning experiences that can be coordinated and applied to enact or reinforce learning.

Table 2-3. Training and education resource considerations (example)

Learning Resource and Sensory Experience Considerations
Abstract	Learning Scale: Individual-Group	Learning Resource (Examples)
Sensory Experience in a Learning Environment	Verbal Symbols	Manuals, Handbooks, Special Texts, Reports
	Visual Symbols	Graphic Designs, Diagrams, Maps, Models
	Still Images and/or Audio Recording	Photograph, Illustration, Symbol, Social Media
	Videography and/or Television	Video, DVD, CD-ROM, Digital Camera, iPad
	Exhibits and/or Static Displays	Posters, Brochures, Advisories, Mockup, Kiosk
	Staff Rides and/or Field Trips	Case Study, Independent Study, Terrain Walk
	Demonstrations	Observation-Participation-Simulations
	Dramatized  Experience	Virtual Vignette-Role Play-Interactive Gaming
	Individual Skills Experience	Military Skills Proficiency Training-Tutorial
	Small-scale Group Experience	Event-Lane-Course EX and AfterAction Review
Concrete	Large-scale Group Experience	Unit Exercise-Capstone Exercise-AAR

As the development phase of the ADDIE process concludes, the milestone plan incorporates any adjustments of the training and/or education action plan for execution. The continuous process of resource analysis and update reflects what resources are allocated to conduct rigorous training and/or education to the approved standard.

The training and curriculum developer, in conjunction with the Threat Manager or OE subject matter expert, confirm that the training plan and/or educational lesson are fully sufficient to present the required conditions in support of the task or action, and in order to achieve the standard. Without this continuum of analysis, the developer risks producing the learning experience that is less than the required robust learning environment and/or beyond capability of the target audience.

Section IV - Implementation

Implementation is the conduct of training and/or education in accordance with the approved action plan. Final preparations confirm that the trainer and/or instructor are prepared and certified to conduct the learning experience. Examples of an action plan for training can be but is not limited to a training plan, gunnery table, or exercise directive. Examples of action plan for educational learning can be but is not limited to a lesson plan, special project, or course program. Implementation includes specific means to collect trainer or instructor assessments to measure achievement of standards and learning outcomes.

The implementation phase is where the learning and training products, processes, and services are actually delivered to the learners. This total package is called a learning platform. A learning platform is an integrated set of services that provide the learners with information (content); activities; resources; and tools such as social learning media that support and enhance their quest to learn new skills and knowledge, and supports the learners' personal learning environment. If the learning platform is delivered in a different manner, such as e-learning or a Job Performance Aid, the platform must still be supported. The OE considerations in this phase are to determine if the equipment and supporting personnel are adequate to ensure the context of the OE is not a distraction or an ineffective enabler to the learning experience. This synthesis of the learning apparatus transitions into the final phase of the evaluation phase.

Figure 2-10. Implementation phase of ADDIE and OEE considerations

Specific assessment means are also ready for feedback to the trainer or educator from the training or education audience, as well as to the training and curriculum developer. Implementation of training and education products and outcomes include ongoing event, program, or course evaluation for continuous improvement. Assessment can include informal comments from a training or education audience, formal questionnaires embedded in an event, and the various levels of after action review (AAR) during and after a learning event. (See figure 2-10.)

The developer obtains assessments and formative evaluation results during implementation. This is a check-on-development to control the quality and implementation of learning products and processes. Data on implementation of OE considerations is used by the developer to sustain or improve quality of conditions in the training implementation packet. Formative evaluations may also lead to new design and development requirements for the OE conditions in the training or education experience.

Management Implementation Requirements

Qualified trainers, faculty, and/or instructors are critical to implementation success. They require appropriate faculty train-up support. They ensure students follow safety, security, and environmental requirements. Local policies and procedures contain additional guidance on implementation of training and education. Responsible and/or accountable leaders, in conjunction with developers, must ensure―

• Continuous internal assessment and formative evaluation.
• Quality control and supervision of the trainer-instructor qualification program.
• Satisfactory institutional services support to the learning experience.

When target audience individuals fail assessments after a designated number of assess, reinstruct, and reassess cycles established for a given learning experience, individuals with substandard performance are removed from the designated program. Proponents follow student dismissal policy and procedures as outlined in AR 350-1.

Ensure all required training and education resources are available for successful implementation. Manage personnel resources and tailor the organizational work force to support implementation. Coordinate implementation operations with all activities involved with the training and education program.

Figure 2-11. Cyber gap vignette–Implementation (example)

Manage facilities, equipment, and supplies. Maintain property accountability for assigned equipment. Ensure equipment and facilities remain in functional/operational conditions. Monitor facilities cleanliness and ensure required repairs. Request materiel in accordance with local supply directives and policy. Ensure instructors receive support, materials, and equipment required for implementation. (See figure 2-11.)

Core Implementation Requirements

Collective, individual, and self-development learning experiences share some common requirements and also have unique requirements for each type of learning experience. Core requirements are to:

• Comply with applicable laws and regulations.
• Comply with safety and environmental protection rules, regulations, laws, and course requirements.
• Ensure proper maintenance of required facilities, material, equipment, and systems.
• Obtain required reference materials and ensure currency of doctrine presented in instruction.
• Train faculty/instructors to implement and manage the training and/or education.
• Ensure implementation is based on approved analysis, design, and development outputs.
• Assess faculty/instructor delivery methods and conscious development of attributes.
• Continuously collect evaluation data per the evaluation plan developed in previous phases and provide appropriate feedback to the proponent.
• Maintain training and education records.

Collective training implementation may have learning experiences than range the cognitive, affective, and psychomotor domains. Characteristics of effective implementation include but are not limited to the following actions―

• Requires performance-oriented demonstration and/or learning to the standard.
• Sustains relevance in accordance with observations, insights, and lessons learned.
• Prepares units, teams, and individuals for missions and essential and/or critical tasks.
• Applies a crawl-walk-run approach to learning.
• Spans live, virtual, constructive, and gaming environments.
• Replicates conditions of a particular operational environment.

The Threat Manager or OE subject matter expert assists in the review and validation process for credible and robust operational environment conditions. Formative evaluation observations and insights are incorporated into the preparation for the summative evaluation in the ADDIE evaluation phase.

Section V - Evaluation

Evaluation is a continuous process that starts during the analysis phase and continues throughout the life cycle of the ADDIE process as well as the life cycle of each learning product. Evaluation includes the quality control mechanism for training and education development and implementation. It is a systematic method to appraise quality, efficiency, and effectiveness of a program, process, procedure and/or product.

The evaluation phase provides a process for decision-makers to determine what to sustain or improve to meet established standards. The formative evaluations that occur during each phase of the ADDIE process support the formal review of a summative evaluation. The two forms of evaluation in the ADDIE review are formative and summative evaluation. (See figure 2-12.)

Figure 2-12. Evaluation phases of ADDIE and OEE considerations

The Threat Manager or OE subject matter expert assists the developer in the review and validation process during informal and formal evaluation processes during this phase of ADDIE. An example of a formal process is the post instructional conference (PIC) at an educational institution. Sustainment and/or improvement of conditions that emerge from such a formal review are stated in academic guidance for subsequent learning experiences. As the ADDIE process is a continuum, the Threat Manager or OE subject matter expert attend sessions that cycle through analysis and lesson or course design review for curriculum update and verification of credible and robust operational environment conditions in support of tasks and actions. Similar formal venues in training environments review and revise learning experiences to confirm conditions to be sustained as well as emergent requirements to be integrated into future learning experiences. Concurrent actions assess and determine how to use limited resources in the most efficient manner to replicate realistic and robust conditions.

Evaluation incorporates the periodic reports from the Army's Quality Assurance (QA) Program. The QA evaluations provide unbiased regular assistance visits, and recommendations. All training and education institutions must conduct evaluations on a continual basis in accordance with policy for the Army’s quality assurance program. Internal evaluations are conducted annually while designated headquarters evaluations are conducted every three years. Specific considerations for evaluation include but are not limited to:

• OE complexities resulting from variable interactions such as noncombatants, cultural and language challenges, media presence, and/or austere infrastructure.
• OE complexities of unified action partners such as governmental and  non-governmental organizations, private sector groups, contractors, host-nation forces, security/police forces, and/or UN peacekeepers.
• Commensurate to their level of education, students understand various combinations of threats employing hybrid strategies (regular and irregular forces) applied against combined arms maneuver and stability operations.
• Doctrinal principles of the operational variables (PMESII-PT) and mission variables (METT-TC) are incorporated into the respective PoIs.
• Training aids, devices, simulators, and simulations (TADSS) replicate OE complexities (training role players, IED simulators, MILES, small arms) to a low-fidelity environment (low fidelity as defined by Army training directives).
• Training ranges and simulation facilities replicate various complexities of the OE (non-combatant targets on rifle range, terrain, key leader engagements, urban or rural sites).
• CoE/institution has identified a Threat Manager or appointed an OE subject matter expert to verify and validate OE content within TADSS and is involved in the ADDIE process.
• Conditions are modified to represent and integrate OE complexities commensurate to the  OE learning spectrum as described in Chapter One.
• TRADOC-approved scenarios or approved derivatives are used for all learning activities that require a scenario.

Use formative evaluations continuously throughout the ADDIE process to check quality of effectiveness and efficiency and make in-progress improvements to learning and learning products. Formative evaluations provide sufficient data to decision makers to make sound, informed training and education decisions. Formative evaluation includes but is not limited to:

• Identify deficiencies/problems/issues rapidly during production (analysis, design, and development phases) to make corrections with minimal cost.
• Evaluate required outputs from each phase prior to expenditure of additional resources.
• Evaluate the effectiveness of the proponent/center/school's staff: managers, administrators, faculty/instructor, training and education developer, and/or any other staff with responsibilities during the ADDIE process, including managing/coordinating efforts with RC, contractors, and/or mobile training teams (MTTs).

Evaluate the quality of instruction, instructional materials, student learning and achievement, and the adequacy of the infrastructure in place to support both resident and non-resident instruction. Critical aspects for individual and collective training to meet its training objectives are that OE and OPFOR complexities set the right conditions within the task-condition-standard framework, and that they provide the appropriate stimulation that drives training unit actions to complete their training tasks and learn, and that such stimulations are relevant and realistic. Also important is the fair and accurate portrayal of potential conditions and a hybrid threat within the context of the established operational environment, while meeting unit training objectives and effecting desired leader development outcomes. An in-depth discussion of the OEE support to evaluation, accreditation, and validation of OE integration efforts may be found in chapter 3, “Operational Environment Enterprise (OEE) Support.”

Experienced training managers and developers use strategies for overlapping processes, spiral development, and rapid prototyping to create quality courses and products while using less time and resources. Due to the ever changing nature of the strategic environment and unit deployments to various operational areas of concern, a training developer may have to use this technique to rapidly incorporate the OE changes to a lesson or training support package. (See figure 2-13.)

Figure 2-13. Cyber gap vignette-Evaluation (example)

Training and curriculum developers overlap elements of analysis, design, and development to improve effectiveness and efficiencies in learning. Information gathered during task and topic development in the analysis phase overlaps learning objective development in the design phase. Completed and approved elements from the design phase may proceed to aspects of development prior to all elements being approved for integration. Implementation is a dynamic action of conduct and concurrent review. Evaluation overlaps every ADDIE phase. Formative evaluation must be conducted throughout the ADDIE process to ensure quality products and credible use of available resources. These are just examples when thinking in terms of non-linear efficiencies. Spiral development, sustainment, and improvement is an iterative continuum of updating and/or adjusting training and education experiences based on formative and summative evaluation data and other guidance from the leadership level accountable and responsible for the Army readiness.