Report of the Program AssessmentTask Force: October 1996

The Program Assessment Task Force was convened in May 1996 pursuant to the Strategic Plan for the University, Beyond 2000: Change. Strategic Goal 5 of the Plan states that the institution should allocate and develop resources on the basis of mission value and performance. Hold the community of students, faculty and staff accountable for the success of the University. (Goal 5, p. 29)

The Plan further specified that the University should establish a University-wide process of program review, needs assessment and opportunity evaluation. (Objective 5.2, p. 30)

To achieve those ends, the Chancellor convened a Task Force that has met during the summer and early fall. This report represents the initial results of that work and should be viewed as an initial report with further elements and refinements to be covered in subsequent reports.

General Principles and Structure of Program Assessment

The purpose of program assessment is to assist the University and its constituent academic units in the pursuit of excellence. The process is intended to

• evaluate program quality and needs
• identify targets and opportunities for program development and improvement, and
• guide the allocation and reallocation of resources.

The system of review that is here proposed involves a cyclical process in which programs will be assessed, establish priorities for the near-term future and receive commitments from the University regarding their future operation. This process is to be under the control and supervision of the Chancellor. The Chancellor shall be assisted in the oversight of the assessment process by an Assessment Monitoring Committee composed of representatives of the University Senate, the Graduate Faculty Council and others designated by the Chancellor. A model charge for that Committee follows:

The Assessment Monitoring Committee will serve in an advisory capacity to the Chancellor in matters pertaining to the Program Assessment Process. Its areas of concern will be to monitor and suggest appropriate modifications to the Assessment Process with respect to the composition Baseline Data provided to units, to the General Guidelines for Self-Study and for External Review, and to the process for selection of External Review Teams. It will also serve as the review body in instances where the parties are unable to reach a consensus regarding the Memorandum of Agreement because of alleged procedural or substantive flaws in the Assessment Process.

The "normal" cycle for Program Assessment will be five to seven years. The Chancellor, in consultation with the Assessment Monitoring Committee will establish and publish a basic but tentative calendar for the review of all appropriate units. The Chancellor also may call the process into action at other than the regularly scheduled interval under special circumstances.

The Assessment Process involves four stages, each of which results in a particular set of outcomes as follows:

1. Compilation and Reporting of Baseline Data

Each unit about to embark on the process will be provided by the University with a five-year compilation of data based on annual reports, budget allocations and expenditures, staffing and space allotments, enrollment and degrees granted, outcomes information from degree recipients and other pertinent information. These data packets should be standardized across the University and will serve to provide a common baseline for the review process. However, each unit also has the prerogative to compile additional data pertinent to its own operation and assessment. One of the responsibilities of the Assessment Monitoring Committee will be to continue to review and revise the composition of this baseline data set so that it will better serve the process. Subsequent stages of the process should be able to rely heavily upon these data rather than having to compile new data independently.

2. Self-Study

The cornerstone of the Assessment Process must be a careful and thoughtful self-study by the unit under review. Such a review must go beyond the description of the current status and operations of the unit to become evaluative, diagnostic and prescriptive focussing on strengths, weaknesses and strategies for change and/or improvement of quality. The objective of the review is to assist units in charting future directions and identifying appropriate performance objectives and its ultimate goal is to guide the allocation of resources to those efforts that will most enhance the quality of the University's academic operations.

A tentative schedule of topics to be addressed in the self-study has been attached as Appendix I. It should be understood as a guide in a general sense and may include many items not appropriate to specific units. Updating and revising this procedural guide will be another of the responsibilities of the Assessment Monitoring Committee.

This process of self-study should be completed within a semester and result in the compilation of a document of approximately 25 pages in length not including appendices for supporting data. The compilation of the self-study should be an inclusive process involving faculty, staff, students, graduates and "clients" of the program being assessed. The final document should be "approved" by unit faculty, and appropriate staff. The self-study then should be combined with the baseline data to comprise the point of departure for the third stage of the process.

3. External Review

Upon completion of the self-study, an external review is to be undertaken. A Review Team of three to five persons is to be convened depending on the size and complexity of the unit being considered. The minimum Review Team would consist of two members drawn from the same field at peer institutions and one member drawn from a cognate discipline or unit within the University. Composition of these review teams should be determined by the Chancellor after soliciting nominations from the unit being reviewed, the Dean of the Graduate School and the Dean of the school or college within which the unit operates and other sources as appropriate. The careful selection of appropriate members for these External Review Teams is critical to the success of the process. In some instances where units or programs undergo accreditation reviews, it may be both appropriate and expeditious to employ the same external reviewers as are involved in the accreditation review. In such cases, the Accreditation Review and the Program Assessment processes would be essentially concurrent. Each Review Team should also have an institutional liaison person from either the Chancellor's or appropriate Dean's office assigned to assist the team in whatever ways it might require.

The focus of this review should be to provide an assessment of the unit that places it in a broader context nationally within its discipline or area. The review should begin by assessing the findings of the self-study. A basic set of guidelines for the Report of the Review Team has been developed and has been attached as Appendix II. It should be understood as a partial inventory inasmuch as the Chancellor or Deans may raise specific concerns that they wish to have addressed in each individual case. Furthermore, the special characteristics of individual units may well augment the most general issues identified here.

The External Review process should take place during the semester following the completion of the self-study. This process ordinarily should take six to eight weeks to complete. The Review Team should be provided with the Unit Self-Study and Baseline Data at least four weeks prior to its Site Visit. The Site Visit would normally last two or three days and be concluded by an "exit interview" providing an overview of its general findings. The final written Report of the Review Team should be completed and delivered to the Chancellor within 30 days of the Site Visit. Copies should be provided to the Chancellor, the appropriate deans and the unit head or director.

4. Discussion of Findings and Memorandum of Understanding

Upon receipt of the Report of the Review Team, the Chancellor will convene a meeting to discuss the findings of the Assessment Process as reflected in both the Self-Study and the External Review. This meeting will involve representatives of the unit, the appropriate deans and the Chancellor. The purpose of the meeting will be to discuss the reports, reconcile discrepancies between them and to develop a final set of recommendations. This meeting should be convened within 30 days of receipt of the Report of the Review Team.

• These final recommendations then will be set out in a Memorandum of Understanding to be developed by the Chancellor. Although this document is to be understood as an informal agreement subject to revision as conditions might warrant, it should convey the outcomes of the Assessment Process. As such, the Memorandum of Understanding should include:
• a summary of the findings;
• a specific list of positive attributes upon which to build, as well as issues or problems to which the unit will be expected to be responsive;
• a specific list of performance targets that the unit will be expected to attain and a schedule by which the targets are to be attained;
• a detailed list of unit or institutional commitments necessary to the solution of identified problems and for the attainment of those performance targets and, finally;
• a tentative schedule for the next Assessment.

The Memorandum of Understanding should be drafted, circulated and signed by the appropriate parties within 30 days of the initial meeting following receipt of the Report of the Review Team. In cases of significant disagreement, the Assessment Monitoring Committee can be called upon to determine if the Assessment Process has been significantly flawed either procedurally or substantively. Consideration by the Committee may be requested either by the unit or by the Chancellor. The Committee will report its findings to the Chancellor within 30 days.

The Memorandum of Understanding will serve as a basic guide to the individual units, schools and colleges and the central administration with regard to policy and budgetary decision making. It should represent a good-faith commitment on the part of both the individual units and the administration to the improvement of the institution by focussing on the achievement of stated objectives. It also will provide a framework for assuring accountability.

Respectfully submitted,

The Task Force on Program Assessment

Appendix 1. Academic Unit Assessment Self-Study Outline

This document is intended as a suggested framework for your unit's evaluation . It may not be detailed enough in some areas, while it may ask for information in other areas that is inappropriate for your unit. Answer each of the following questions, as appropriate, in the context of your unit's Academic Planning Baseline Data, which you should attach as Appendix 1. In your responses, please be interpretive rather than duplicative and refer to supporting documents wherever possible.

A. Unit Description

1. Describe briefly:
   1. the instructional offerings associated with the unit, including both graduate and undergraduate degrees and certificates.
   2. the areas of scholarly productivity and creative performance associated with the unit.
   3. the outreach, service, and clinical activities of the unit.
2. What is the unit's role within the University, the State, and the Region? Comment on similarities to and differences from any other programs within the state or region that offer related training.
3. How is the unit organized? Describe the unit's administrative and governance structure. Include its position within the University administrative structure.
4. List the names and titles of the faculty and professional staff of the unit. Include an up-to-date curriculum vita in Appendix 2 for each person listed.
5. Identify five peer units at other universities and five that provide targets of aspiration for this unit. How does this unit compare with others nationally? What evidence suggests this conclusion?
6. List the supporting documents on file for this review.

B. Unit Mission, Goals, and Recent History

1. What are the major goals of the unit? How have these evolved through recent years with respect to the unit's mission statement? How are they expected to change in the future?
2. How do these goals relate to the strategic plan and mission of the University as expressed in the University of Connecticut strategic plan, Beyond 2000: Change?
3. Describe the process for reviewing the unit's strategic plan and assessing its achievements and goals.
4. Provide a brief description of the major changes that have occurred since the last program assessment or within the past five to seven years. Attach the summary from the most recent program assessment. Comment on how the issues raised therein have been addressed.

C. Faculty and Staff

1. Describe briefly:

1. the profile of full-time and part-time faculty, including graduate assistants in the teaching programs of the unit. How are these instructors assigned to their teaching loads and particular courses? Do part-time faculty members participate in the academic program's meetings and discussions, in curriculum planning, and/or in course design? How are part-time faculty and teaching assistants identified, and how are their credentials evaluated?
2. the profile of full-time faculty, professional staff, and graduate assistants in the graduate education, research and creative performance programs of the unit. How is this profile reviewed?
3. the contributions of unit members through professional service (both external and internal), outreach, and clinical activities.

2. What is the gender and race/ethnicity composition of the unit? Describe the major features of the program's affirmative action plan and its implementation, including efforts underway to attract and retain women and minority faculty.

3. Do graduate students in your unit serve as TAs in other programs on campus? Are graduate students in other programs employed in your unit?

4. Are orientations, workshops, or seminars conducted to improve the teaching effectiveness of TAs? How is the teaching performance of TAs evaluated?

5. Describe the classified staff in the unit. What has been the turnover rate in these positions during each of the previous five years? If high, what steps have been taken to uncover and address the problem(s)? What changes are underway or contemplated to strengthen the staff support for the unit's activities?

D. Scholarly Productivity and Creative Performance

1. Referring to the Academic Planning Data (Appendix 1), evaluate the level of scholarly activity in the unit. Address the quality and quantity of the unit's publications, presentations at academic and/or professional forums, and performances, as appropriate.
2. Referring to the Academic Planning Data (Appendix 1), evaluate the level of internal and external funding for research, performance, or creative activity in the unit. Is the unit competing effectively for external support?
3. Describe any significant research interactions with external entities (public or private) developed by the unit. What have been the benefits of these interactions and the drawbacks, if any? How do they contribute to the unit's research goals?
4. Briefly describe how the research, performance, or creative activity in your unit compares nationally (as defined in section A.5.) with respect to these activities.
5. What does the unit do to encourage and develop research, performance, or creative activity? Describe any deficiencies in facilities and resources which negatively affect the unit's attempts to reach its research objectives.
6. What is the balance of scholarly depth and breadth in the faculty, and what is the balance of traditional views with work taking place at the field's frontiers?
7. Describe and evaluate the unit's participation, leadership, and influence in the academic profession through such avenues as professional associations, review panels, and advisory groups.
8. Describe the unit's potential for responding to changing directions and new external opportunities. What indicators show the level of morale, commitment, and sense of continuing self improvement of the unit?

E. Undergraduate Programs

1. Describe the unit's role in college and University general education programs.

1. What has been the planning process for these offerings and how are they coordinated with other general education courses, and with courses offered for students majoring in the field.
2. What proportion of these courses is taught by full-time faculty, part-time faculty, and graduate students? How is the quality of instruction assessed? Where graduate assistants have a primary instructional role, how are the supervised?

2. With respect to the undergraduate major(s) curriculum and courses:

1. How does the undergraduate major reflect the basic goals of the academic program? What evidence is available to compare the curriculum with that of similar programs nationally (as defined in section A.5.)? Does an accrediting body prescribe any portion of the curriculum? If so, how?
2. How are courses in the undergraduate major(s) coordinated? What evidence is there of sufficient breadth and depth of course offerings, as well as balance among the various specialties to meet student needs and interests?
3. What specific efforts are made to incorporate new knowledge and perspectives into the curriculum? What efforts are made to involve students actively in their learning through internships, research projects, seminars, independent study, studio courses, etc.? Describe any innovations added to instructional programs since the last review.

3. Concerning the undergraduate major cohort:

1. How does the quality of students in the major compare with the quality of students in related fields at the University of Connecticut? Referring to the Academic Planning Date (Appendix 1), discuss entry test scores, average entering G.P.A., and placement test scores. Does the unit have admissions requirements or other prerequisites for entry into the major? If so, what are they?
2. Explain any significant changes in the undergraduate major enrollment and in degrees granted since the last review. What are the implications of the average lengths of time required for degree completion? What are the retention rates for the undergraduate majors? Explain low enrollment or low graduation rates, if below DHE criteria. What efforts are underway to improve performance in these areas?
3. What indicators does the unit utilize to monitor the success of its graduates? How does the quality of these graduates compare with student quality in this field nationwide (as defined in section A.5.)? Describe any honors or awards received by the unit's graduates.
4. Referring to the Academic Planning Data (Appendix 1), comment on the quality of the unit's efforts to attract, support, and graduate traditionally underrepresented ethnic or gender groups. How does the proportion of degrees granted to women and minorities compare to the University as a whole and to the field nationwide? What specific plans and programs does the unit have in place for increasing the proportion of graduates from traditionally underrepresented groups? Include measures taken to provide special advice and support for such students while enrolled.
5. What is the unit doing to serve nontraditional and part-time undergraduate students?
6. Describe the process and structure of the undergraduate advising in the unit. What is the quality of advising for undergraduate majors? How has the advising process been evaluated?

F. Graduate Programs

Address each of the following topics for both the master's and doctoral levels, as appropriate for the graduate degrees offered.

1. Overview of the graduate program(s):
   1. Describe, in general terms, the graduate program(s) offered by the unit. How do the graduate programs reflect the basic goals (Section B) of the academic program?
   2. What evidence is available concerning the quality of the unit's graduate program(s)? How is this information used to strengthen the graduate program(s)? Describe in particular extramural awards and grants for the purpose of graduate education, either to individual students or to the program.
   3. What changes in the program(s) have occurred since the last review? What changes (especially innovations) are now underway?
2. With respect to the graduate curriculum and courses:
   1. What evidence is there of sufficient offerings and balance among the unit's various specialties? Is there sufficient breadth of course offerings and sufficient depth for specialization? How are the courses in the graduate program coordinated? What plans are underway to modify the graduate curriculum in light of available information?
   2. What evidence is there of whether the courses meet student needs?
   3. In what ways besides individual thesis or dissertation research are students involved actively in their learning? For example, through internships, practica, and/or graduate assistantships?
   4. Do students have adequate resources to carry out their studies (e.g., library, office and lab space, supplies, equipment, travel, photocopying, etc.)? What additional resources would be required to improve the quality of the graduate program substantially?
3. Concerning the graduate student cohort:
   1. Assess the quality of the graduate student cohort, based on collective GRE or other test scores, collective grade point averages, or other admissions criteria. How does the quality of students in the graduate program compare with student quality in other similar programs? How does the quality of current students compare to the students in the program since the last review? Base the answer on objective and anecdotal data.
   2. What is the current gender and race/ethnicity composition of the graduate student cohort? How do these figures compare with those for undergraduates within the unit and for similar graduate programs at other schools? What efforts are underway to attract and retain well qualified students from underrepresented groups? What mechanisms are used to support these students?
   3. What is the current composition of the graduate student cohort with respect to geographic origin? How do these figures compare with figures for similar graduate programs at other schools (as defined in section A.5.)?
   4. What mechanisms are used to recruit students? Is the program competing well for top students?
   5. Are stipend levels adequate? Is stipend availability adequate? In addressing this, consider how many of the total number of graduate students have a teaching or research assistantship (both full and half). Also address other forms of support available to graduate students (e.g., predoctoral fellowships). What sources and amount of funding is available for summer support?
   6. What is the nature and quality of the advising for graduate students. and how has advising been assessed? What is the average ratio of student/faculty during thesis and/or dissertation supervision?
   7. What are the retention rates in the graduate program for both master's and doctoral students? What is the average time to degree completion? How do these figures compare with those from the last review?
   8. Explain any significant changes in graduate student enrollment and in degrees granted since the last review. How do these figures compare to peer programs, as defined in section A.5.? Explain low enrollment or low graduation rates, if below DHE criteria. What efforts are underway to improve performance in these areas?
4. Describe the extent to which this unit interacts programmatically with other units (both within and without the university) at the graduate level. Cite other units where students frequently take coursework or other program options in your unit. List courses in your unit which are frequently taken by students within other graduate degree programs.

G. Student Outcomes Assessment

There is growing national awareness of the need to articulate clearly what skills and content students should expect from their post-secondary education, and to assess how effective programs are at giving students those skills. Answer the following questions for both the undergraduate and graduate cohorts, as appropriate:

1. Describe what the graduates of the program should know and be able to do when they leave the university, and how the unit measures or otherwise assesses actual student achievement. How do you use assessment of student learning outcomes to make curricular offerings more effective at meeting the goals set for the students?
2. Specify how student outcomes are related to the mission and goals of the unit, the college (if appropriate), and the University.
3. Describe how the data are collected to ensure reliability and validity. For example, are the data collected from a representative sample of students?
4. How will the results of the student outcomes assessment be incorporated in strategic planning and curricular review process?
5. Provide a list of all graduates from the last three to five years (both graduate and undergraduate) and indicate to the extent possible where they have been placed. How do graduates of the program view their experience, and how are their views solicited? What program modifications do these views suggest?

H. Outreach and Public Service

This term refers to educational efforts, leadership, and sharing of knowledge off-campus, for example in the local community and throughout the State and beyond.

1. Describe the nature of academic outreach and public service activities in the unit.
2. How do these activities reflect the goal(s) described in Section B, and the particular needs of the state and region?
3. What evidence is available to document the quality and effects of these activities?

I. Collaboration with Other Units

1. What are the other departments, schools/colleges, and centers with which the members of the unit collaborate most frequently? Describe the nature of those efforts and an assessment of successes and disappointments.
2. In what ways (if any) do disciplinary or unit boundaries inhibit or enhance the ability of the unit to develop new approaches to research, grant competitions, teaching, or service?
3. Under ideal circumstances, what kinds of collaborations (e.g., research and teaching, grants, alumni programs) with other units would be desirable?
4. What opportunities are there for carrying out interdisciplinary research projects with other units on campus or with other universities or agencies? How successful is the unit in accommodating these needs? How desirable are these kinds of interdisciplinary relationships?

J. University-wide Support and Facilities Services

1. Describe and appraise the current institutional support services and facilities for the unit's:
   1. teaching programs;
   2. research, creative production, or other scholarly activities;
   3. recruiting of students, faculty, and staff;
   4. outreach, including professional and community service;
   5. administration.
2. Rank order the unit's specific and most pressing resource needs (e.g., library, laboratory, classrooms, classroom support, office personnel, research assistants, others)?

K. Summary Assessment and Future Directions

1. Summarize the major strengths and weaknesses of the unit and the problems it faces in the foreseeable future.
2. The quality of an academic unit can be assessed in many ways. In terms of the three criteria below, describe the overall quality of the unit.
   1. a. Resource criteria (e.g. student selectivity or demand; faculty prestige, training, and teaching loads; grants and contracts; library; equipment; and support staff).
   2. Reputational criteria (e.g. national or international ranking, or other judgments of the program's students, faculty, resources, and productivity).
   3. Outcomes criteria (e.g. faculty scholarly productivity, awards and honors, research contributions, teaching performance, service to state and nation; student gains in knowledge, students' professional achievements, placement personal/or career development, program alumni opinion).
3. In what areas has the unit improved or deteriorated since the last review? Describe the evidence used to support the conclusions.
4. Describe new directions in curriculum, resources, research, reorganization, staffing or student clientele planned for the next few years aimed at strengthening the unit, in conjunction with your strategic plan.
5. What plans are underway to capitalize on individual faculty strengths and to overcome weaknesses?
6. With suitable incentives, are there new ways that the unit could enhance programs for undergraduate and graduate students, for research, service, and University operations?
7. What improvements in quality might be projected if additional resources were available, and what would be their expected effects?

L. Executive Summary

In no more than two pages, highlight the most salient points of this self-study. Place particular emphasis on new directions and remediation of existing problems.

APPENDIX II General Outline for External Review

The general purposes of the external review portion of this assessment process are to corroborate the findings of the unit's self-study and to help place that assessment in a broader context relative to other programs in the unit's discipline or area. The following list of questions is to be provided to the review teams as a basic skeleton around which to frame their comments. It is presumed that additional questions pertinent to individual programs will be raised at the outset of each external review.

1. Has the unit employed appropriate measures to assess the quality and scope of:

• its scholarly productivity?
• its instructional programs?
• its outreach and public service efforts?

2. Summarize any significant differences in your assessment of the unit's performance and potential and those contained in the unit's self-analysis in terms of:

• its scholarly productivity.
• its instructional programs.
• its outreach and public service efforts.

3. How would you assess the unit's effort and reputation as compared to peer units within its own field?

4. Does the plan articulated by the unit for the next five years set attainable targets with regard to:

• its scholarly productivity?
• its instructional programs?
• its outreach and public service efforts?

5. Provide additional observations or alternative directions that you think appropriate to be considered in the assessment of this unit.