Program Evaluation in Distance Education

朱琳

(Sias International University, 168 East Renmin Road, Xinzheng, Henan, The People's Republic of China (PRC))

【Abstract】In this fast paced world, many people do not have enough time to sit in a classroom and listen to a lecture by a professor for three to four hours. Distance education has become an important way to learn and teaching online is a basic skill for the teachers and trainers. Recent reports have revealed that over 20% of college students are enrolled in at least one course delivered at a distance.(Simonson M, Smaldino S, Albright M & Zvacek S,2010) About the quality of distance education, evaluators need to do a lot of researches that determine whether our education is useful or not. This article introduces the concept of program evaluation in the distance education, the history of educations evaluation, program evaluation procedures and categories, program evaluations approach (the AEIOU approach) and the great importance of program evaluation in distance education.

【Key Words】program evaluation; distance education

【中圖分類號(hào)】G64.23 【文獻(xiàn)標(biāo)識(shí)碼】A 【文章編號(hào)】2095-3089（2016）13-0-02

What is program evaluation?

Evaluation is an essential component of program improvement and renewal and long-term success. Willis (1993, p. 70) cautioned that ‘‘even the best designed or adapted distance delivered course will likely require revision. Moreover, distance education program evaluation results can be extrapolated to improve other distance education programs. According to Verduin and Clark (1991, p. 184):Good evaluation will also assist distance educators in thinking about what they are trying to do and achieve as they implement programs and activities. Continued development in distance education is essential and evaluation can reveal what is effective and what is not. Program evaluation is the systematic investigation of the worth of an ongoing or continuing distance education activity (Joint Committee on Standards for Educational Evaluation, 1994). Martinez, Liu, Watson, and Bichelmeyer (2006) discussed the importance of evaluating distance education programs. Evaluation of program is used to identify strengths and weaknesses as well as the benefits and draw-backs of teaching and learning online. (Martinez et al.,2006). Program evaluation as the basis of verifying the progress of objectives and strategies now, is going to act as the most important part of education process. Thus, it reflects how the education strategy worked and which part makes the objective failed or completed.

Overview of history and theory of program evaluation

History of program evaluation is divided into three periods: sprout period, formation period and development period. Sprout period: during the second half of the nineteenth century, western countries made a quantitative research on individual intelligence differences by inspection and examination on basis of experimental psychology and mathematical statistics, namely exploration on students' learning objectives, standardized measurements, which laid the foundation for education measurement theory. At the beginning of the twentieth century, Thorndike's" the introduction of psychological and social measurement" was regarded as the epoch-making masterpiece in the measurement movement history of the U.S. Thorndike believed that every existing entity has quantity, and that any entity that has quantity can be measured. Along with the development of education surveys, assessment on school education by using the measurement results of student's learning ability entered into the budding period.

Formation period: around 1930, the U.S began to lay emphasis on the education of developing personal talents and personality and nominated Taylor to host the ‘curriculum and assessment research, which was known as" eight years of research". The basic guiding principle on education assessment of Taylor's Committee was: based on the education goal and education effect assessment program, take a man's all-round development as the main targets and pay attention to the value of education effect, analyze the education objective and make it specific. Taylor thought that the education assessment was designed to measure the school education activities to achieve the goal of education (approximation). In 1949, the United States established the National Education Assessment Committee, which was designed to certify whether a college or professional education program has reached predetermined qualifications and standards.

Development period: after the Second World War, education assessment was paid more attention to and the research on foundation of education assessment theory and method research were strengthened. Along with the education reform, the direction of educational assessment and educational measurement switched from the study of individual differences as the center and the purpose of selecting talented students from education to equal opportunity in education, and caring and helping each student's all-round development. The contents of education evaluation objects developed from individual assessment( curricula, teaching materials, teachers ) in the teaching process to comprehensive assessment including the whole education system ( decision making, planning, management, system etc.).

Program evaluation procedures and categories

A new distance education system is being planned and implemented. There is considerable concern that the time and effort is required to move to distance delivery produced a valuable educational experience, thus, evaluation is regularly a part of plans to move from traditional face- to-face instruction to distance education. Kirkpatricks (2006) evaluation approaches with its four levels of evaluation as shown below, supplemented by Phillipss (2003) fifth evaluation level- return on investment (ROI)----seems to be the preferred approach for many trains and some educators.

Level 1-Reactions (Did they like it?)

Level 2-Learning (Did they learn it?)

Level 3-Transter (Will they use it?)

Level 4-Results (Will it matter?)

Level 5-Return on Investment

Kirkpatricks and Phillipss evaluation approaches distance education works academically to produce required achievement gains. The evidence is clear that students learn just as effectively when they are taught at a distance as compared to when they learn knowledge in a traditional classroom. (Simonson, 2002). According to Woodley and Kirkwood (1986), six categories of evaluation information could be collected from distance education activities.

1.Measure of activity

2.Measure of efficiency

3.Measure of outcomes

4.Measure of program aims

5.Measures of policy

6.Measures of organizations

Six categories of evaluation are not used for every distance education activity. Certainly, some modest evaluation activity is almost always necessary. It is important that the activities of evaluators are matched to programmatic need. Woodley and Kirkwood (1986) have summarized evaluation in distance education as being a fairly eclectic process that utilizes procedures that should match program needs to evaluation activities.

The AEIOU approach

Fortune and Keith (1992) proposed the AEIOU approach for program evaluation, especially the evaluation of distance education projects. The effectiveness of this approach has been demonstrated through evaluating the activities of the IOWA Distance Education Alliance Star Schools Project (Simonson & Schlosser, 1995a; Sorensen, 1996; Sorensen & Sweeney, 1995,1996,1997, Sweeney, 1995), a multiyear, statewide distance education activity.

The AEIOU approach is similar to Woodley and Kirkwoods in that it is an eclectic one that uses quantitative and qualitative methodologies. It has two primary purposes as an evaluation strategy. First, the model provides formative information about the staff about the implementation of their project. Second, it provides summative information about the value of the project and its activities. The AEIOU evaluation process provides a framework of identifying key questions necessary for effective evaluation.

Component 1- Accountability (A)

Did the project planners do what they said they were going to do?

Component 2-Effectiveness (E)

How well done was the project?

Component 3- Impact (I)

Did the project, course, or program make a difference?

Component 4- Organizational Context (O)

What structures, policies, or events in the organization or environment helped or hindered the project in accomplishing its goals?

Component 5- Unanticipated Consequences (U)

What changes or consequences of importance happened as a result of the project that was not expected?

The AEIOU model is a dynamic one that permits the evaluator to tailor the process of program evaluation to the specific situation being studied. (Simonson, M. Smaldino, S. Albright, M., & Zvacek, S, 2010).

The great importance of program evaluation in distance education

Evaluations resulting in program refinements are highly important if quality is to be achieved and maintained. Galbraith, Sisco, and Guglielmino (1997, p. 113) stated, ‘‘Evaluation is an essential element in effective programs for any target group. Scriven (1981) reported that a common use of program evaluation was for accountability purposes. He also identified other uses to include (a) determining program effectiveness, (b) identifying program weaknesses to enable administrators to improve program effectiveness, (c) providing evidence of effectiveness to doubters, and (d) providing information that can be used for program renewal. Keegan (1996) recommended that the evaluation of any distance education program should focus on four aspects: (a) the quantity of the learning (i.e., enrollment, new learner markets, and course-completion rates), (b) the quality of the learning (i.e., effectiveness of courses or program to enable desired learner outcomes), (c) the status of the learning (i.e., transferability of coursework and employer recognition of degrees and certificates), and (d) the relative cost of the learning (i.e., institutional cost effectiveness and cost benefits). Alfred P. Rovai (2003).

Reference:

Galbraith, M., Sisco, B., & Guglielmino, L. (1997). Administering successful programs for adults. Malabar, FL: Krieger.

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.

Martinez, R., Liu, S., Watson, W., & Bichelmeyer, B. (2006). Evaluation of a web-based masters degree program:lessons learned from an online instructional design and technology degree program: Lessons Learned from an online instructional design and technology program. Quarterly Review of Distance Education, 7(3), 267-283.

Scriven, M. S. (1981). The logic of evaluation. Inverness, CA: Edgepress.

Campbell, J. O., Lison, C. A., Borsook, T. K., Hoover, J. A., & Arnold, P. (1995). Using computer and video technologies to develop interpersonal skills. Computers in Human Behavior, 11(2), 223–239.

Simonson, M. (2002). In case you are asked: The effectiveness of distance education. Quarterly Review of Distance Education,3(4), 33-36. .

Woodley, A,&Kirkwood;,A. (1986). Evaluation in distance learning. Paper 10. Resources in Education. (ERIC Document Reproduction Service No.ED304122)

Simonson, M. Smaldino, S. Albright, M., & Zvacek, S. (2010). Teaching and Learning at a distance. Evaluating Teaching and learning at a distance. 34,73-76.

Verduin, J. R., & Clark, T. A. (1991). Distance education: the foundations of effective practice. San Francisco, CA: Jossey-Bass.

Willis, B. (1993). Instructional development for distance education. Syracuse, NY: Syracuse University.