Student Learning

Contents

• Foreword
• Statement of Questions Addressed
• Methodology
• Findings, Interpretations, and Analysis
• Discussion, Recommendations, and Conclusions

Editor's Note: The most challenging of the reports found in this self-study probably is the report on Student Learning. The University has found itself engaged in several series of ongoing dialogues about the language of assessment that has been introduced into the processes of evaluating both purposes and performances of institutions of higher education throughout the country. There is little question that Student Learning is central to the mission and activities of faculty, staff, and students at Cal Poly. There are questions about exactly how we will define, talk about, and make use of the tools of assessment. There are questions about whether this new language is really new, or perhaps new words to describe attitudes and processes already long in place at this university. The Steering Committee agreed to provide the following "Foreword" in order to provide a historical backdrop to the real difficulties with which the Subcommittee on Student Learning had to deal.

Foreword

During the 1990s Cal Poly, like many other colleges and universities has had to re-examine traditional attitudes toward faculty and teaching relative to new mandates for greater focus on students and their learning. This re-examination has caused controversy. Some have claimed the privileging of learning over teaching as a paradigm shift for higher education. Some reject those arguments as facile polemics. Many outsiders view faculty skepticism as resistance to accountability. Many faculty members hear the current discourse of student learning (objectives, goals, and assessment) as symptomatic of the same syndrome that has compromised K-12.

Evidence that Cal Poly has been, and continues to be, engaged in rethinking what it means to organize itself around learning is found in a broad array of university initiatives and in the evolution of academic planning processes on this campus. "Student learning outcomes" were hardly unknown at Cal Poly in the 1980s. Indeed, starting about the mid 1980s, then general education programs at Cal Poly and the other CSU campuses were premised on student learning outcomes articulated by the system-wide Academic Senate. Additionally, the Campus Administrative Manual (CAM) detailed common student learning outcomes to be evidenced in all Cal Poly students' senior projects. Technical and professional programs answered to their specific accreditation agencies and to Cal Poly's Senate Program Review Committee. Academic programs answered to the same Program Review Committee and occasionally to external reviewers. The university answered, as a whole, to WASC. Nevertheless little sustained accountability for student learning inhered in those processes and relationships. For example, Cal Poly never assessed student learning in its GE program. Nor did it review senior projects in any systematic way for evidence of the learning outcomes detailed in CAM. An examination of Cal Poly's program review guidelines and templates since the last WASC Self-Study (1989) clearly reveals an evolving change in the focus on learning in the majors as well. The differences between 1989 Cal Poly Program Review Guidelines and the current (1997-1998) Cal Poly Program Review Template are important as indicators of change in this university's criteria for a successful academic program and because they point to a cultural shift in progress at Cal Poly.

Although the implementation of change across the campus's programs has been uneven and the merits of the actual change itself seriously disputed, the change is significant. The 1989 Guidelines inquire about program enrollments, faculty, curricula, and advising, but the only direct measure of student learning even mentioned is a question about the professional employment and advanced education of program graduates. Indeed, a request for one notable program accomplishment in five years is followed quickly by another for ten faculty accomplishments during the same period. There is no particular mention of student learning and accomplishments at all, except in the query about employment and advanced study for graduates. In contrast, the 1997-98 Program Review Template identifies "instructional issues," including "educational goals" and "intended student outcomes," "program content and skill coverage," "instructional design and methods" including "innovations," and "assessment methods and data" as the most substantive issues, comprising well over half of the program's self-study and report.

Of the topics included under the "instructional issues" rubric, "assessment methods and data" is the most fully developed. This section distinguishes assessment of student learning at the course and program levels as distinct from program evaluation by alumni or professional advisory boards and program evaluation based on placement data for graduates, student participation in research, and faculty scholarship. The implications of the latter are not completely valid indicators of learning. In affirmation of its new emphasis and after completing the reviews scheduled for 1997-98, the Academic Program Review and Improvement Committee sent a memo to deans and department chairs urging them to work harder to do the following:

1. sharpen the focus in program mission statements;
2. clarify specific high-priority learning outcomes at both program and course
3. link more systematically program mission, goals, outcomes, with program planning;
4. consult more systematically with other professionals regarding program design, delivery, and improvement; and
5. gather and use student and alumni feedback for course and program improvement.

One committee's actions, no matter how radical, hardly constitute a cultural change. However the same differences between the 1989 program review guidelines and the 1997-98 program review template are also documented in other committee and university processes and reports, as well. The Senate's curriculum proposal templates for courses and programs evidence a parallel development during the same period. A sample completed course proposal from 1986 is all of two and a third pages with only a few bullet points each under Expected Outcomes, Methods of Instruction, and Methods of Evaluating Outcomes. In contrast, the template, itself, (absent any course specific information) for major, support or elective course proposals in the current curricular cycle runs to six pages, which, unlike earlier forms require faculty members to do all of the following:

1. provide a substantive one-page description of what they expect students to learn in the course;
2. provide an example of planned course content, emphasizing how it will support the stated learning objectives;
3. explain the necessity of any mandatory prerequisites or co-requisites listed for the course;
4. identify each of the modes of instruction or teaching strategies to be used in the course and explain how each is the most effective way of conveying the material to be taught in that component of the course; and
5. identify, in order of importance, the primary methods to be used for assessment of student learning, explain how each method of assessment will establish whether the student has achieved the learning objectives in Section III, and prepare a table or narrative showing the learning objectives and the methods used to assess each objective.

For general-education course proposals, the current template requires all of the above plus summary sheets indicating how the proposals address specific GE Program Learning Objectives and Criteria. They must elaborate on how content, pedagogical methods and assessment methods are all best suited to these Learning Objectives and Criteria. They must also document for every course a writing component, the extent of which varies according to GE area. Assuming they are linked, curricular and program review processes are central to academic planning and improvement at most universities.

The university developed its first Strategic Plan during the early 1990s, partly in response to its last accreditation review. Although it was amended in the middle 1990's, the substance remained much the same except for new interpolation of commitments to more global/international education. Three sections of that plan are relevant to the evolution of this university's current engagement with student learning.

The first is the mission statement, which was written in the late 1980s before any other portion of the plan. It is notable for its privileging of faculty activity over student learning:

The second is the section on Student Satisfaction, which emphasizes the importance of a service orientation in student services and administrative processes. It also speaks about cross-constituent community and respect for individual rights and differences. However, as the only section of the plan directly focused on students, it does not say much about learning. The implication is not that student satisfaction should be the primary or exclusive measure of student learning, but rather that the section devoted to students does not address learning per se. The third, and, perhaps, most relevant section is the one on Academic Programs. It goes further--to assert that "Cal Poly's academic programs . . . should all be assessed periodically to ensure that they meet student and societal needs." However, the academic program section of the plan says nothing about assessing either GE or major programs with respect to student learning goals and outcomes. It does tie program size to program quality among other variables but "quality" is never defined. Both the section on academic programs and the other sections of the plan assert a number of principles and mandates that apply to educational processes but are not expressed as student learning goals or outcomes.

Cal Poly's Strategic Plan provided a comprehensive framework designed to unite disparate campus units around a mission, some goals, some principles, and some specific actions. However, its elaboration of an academic vision and plans for the implementation or refinement of that vision were quite broad and needed further development. Therefore, in 1994 a special task force comprised largely of faculty but also including representatives from other areas went to work to articulate the university's intentions with respect to academic programs in more detail. The October 1995 Report of Cal Poly's Curriculum and Calendar Task Force, also known as Visionary Pragmatism, is the first comprehensive university-wide academic planning document for the university and it's the first university plan to position student learning so clearly at the center of everything else. The very membership in the Task Force of two representatives from student affairs, a representative from the library, a student from ASI, and a staff member with curricular responsibilities from the Academic Programs Office suggested a new cross-functional consciousness of what it might mean to organize a university around learning.

The official charge to the Task Force from senior administration in consultation with the Academic Senate Executive Committee was to "establish principles for baccalaureate programs across the campus, constructing a template within which the programs will revise their curricula, integrating the co-curriculum with the baccalaureate degree, and guiding the process of change in curriculum and calendar." The Task Force's Report begins with a list of characteristics of a Cal Poly graduate. Other than the Senior Project Learning Outcomes detailed in CAM and the General Education Outcomes detailed by the CSU system-wide Senate, their list of attributes for Cal Poly graduates is the first university-wide set of learning goals (some would say outcomes) for all Cal Poly students.

The next and longest section of Visionary Pragmatism defines six major curricular goals along with operational guidelines governing learning outcomes, teaching and learning processes and practices, learning environments, and general education. The fist goal explicitly asserts the centrality of learning: "Cal Poly's primary goal is to enhance learning. This goal should permeate all of the following goals." The fourth goal implies a connection between program evaluation and learning goals: "Curricular and instructional practices should be effective in terms of attaining their designated learning goals." Visionary Pragmatism is important in Cal Poly's evolution toward a more direct focus on student learning, not only because of its explicit assertions of the importance and centrality of student learning and the inclusion of multiple constituencies in its creation, but also because of its integration of affective, social, physical, and cognitive development in its integration of the previously disparate domains of student and academic affairs. Visionary Pragmatism goes further than any previous university blueprint for change to challenge the traditional vertical organizational structures characteristic of Cal Poly as well as most other universities in the past in favor of a holistic vision of student development.

Finally, at least one more university-wide initiative attests to the centrality of student learning in the current Cal Poly environment. Beginning in 1994-95 and continuing into the present, Cal Poly has experimented with a supplemental funding mechanism known as the Cal Poly Plan (PDF). Since the plan is premised on additional fees from students, some might say it inhibits student learning. However, the plan must be viewed in relationship to parallel efforts to increase university support from the private sector (the Centennial Campaign) and a simultaneous effort to increase institutional productivity to release more resources for quality improvements in student learning. In that larger context, students, employees, foundations, employers, alumni and other external constituents are all being asked to share in the responsibility for maintaining and improving the quality of student learning at Cal Poly. In short, the Cal Poly Plan is designed as "a model for how the State of California can meet future demand for public higher education from its citizens in a time of dramatic enrollment growth, rising public expectations for quality and efficiency, and limited public resources." Cal Poly students have a very large say in deciding whether to impose additional fees and, if so, what the amounts of the increments will be. They also have equal representation on a Cal Poly Plan Steering Committee (comprised of three students, three faculty, three staff, and three administrators) that creates Cal Poly Plan policy and monitors spending of Cal Poly Plan dollars. All projects and positions funded by Cal Poly Plan dollars must directly benefit students. The official goals of the plan are to decrease student time to degree, increase student learning, enhance institutional productivity and productivity in teaching and learning, promote more effective use of fixed resources and implement comprehensive assessment and accountability procedures. All project proposals must address these goals, outline budgets and timelines, and detail assessment plans for the projects. Progress and final reports require the inclusion of assessment data as well as summaries of actual expenses, timelines, etc. Complete proposals and reports are available for review in the university's Institutional Planning and Studies Office.

Insofar as assessment is crucial to continuing improvement of student learning, it is fair to say that the university has approached better and more systematic assessment through 1) this decade's changes in curricular and program review processes and 2) the Cal Poly Plan proposal and report processes. The extent to which Cal Poly Plan funded projects evidence valid student learning assessment is apparent in the Cal Poly Plan proposals and reports. The extent to which academic and student affairs programs evidence valid student learning assessment should be apparent in the next section of this chapter.

Top

Statement of Questions Addressed

We have taken our Subcommittee's charge directly from Appendix A of Provost Zingg's letter of August 28, 1998 to Dr. Ralph Wolff, Executive Director of WASC, included four questions intended to focus Cal Poly's self-study of its intellectual environment. The first of these questions served as the point of departure for the efforts of the subcommittee on Student Learning:

• To what extent is Cal Poly focused on student learning, accountable for student learning, and committed to continuous improvement in student learning?
• What additional actions are appropriate to advancing these goals?

We began our efforts to answer these and other pertinent questions by debating the theoretical and practical implications of this charge. Our initial concern was whether to collect new empirical data or to use existing materials. Our decision was influenced in part by the difficulty of constructing new survey instruments in a short time as well as by the tremendous diversity of Cal Poly's programs. This diversity precluded the construction of a "one-size-fits-all" survey instrument. Furthermore, we felt the collection of new data would not necessarily reflect the assessment of student learning as it has been actually practiced at Cal Poly in the recent past. Consequently, we opted to use existing program review and accreditation materials on campus.

Our approach in reviewing these materials was guided by the unique goals identified by each department or program for itself. However, in order to introduce some standardization to the process, we agreed to develop a template reflecting commonly recognized standards in the field of learning assessment. (See the section at the end of this report for a discussion of the potential inapplicability of this approach to some programs.) Thus, while the present report is in large part a summary of the state of student learning assessment at Cal Poly, we have also included opinions regarding outstanding practices in assessment as well as recommendations for how other practices may be improved.

Top

Methodology

This study, focusing on student learning at Cal Poly, utilized document analysis and program interviews in order to address the subcommittee's research questions. The documents studied included academic department program reviews, external reviews, accreditation reviews, and annual program reports.

The size and complexity of the university as well as time and resource constraints made necessary a sample of university programs, rather than a census, in order to report an overview of student learning campus-wide. The sample was comprised of twelve different categories. They included: the College of Agriculture, College of Architecture and Environmental Design, College of Business, College of Engineering, College of Liberal Arts, College of Science and Mathematics, University Center for Teacher Education, Cal Poly Plan programs, Interdisciplinary Studies, Student Affairs, Experiential Education, and the Library.

In an effort to report on programs serving the highest number of students on campus, the two largest academic programs were selected for study among each of the colleges. The remaining programs we studied represent the diversity of programs that have a sizable impact on student learning within the university. In order to make the subcommittee's research task manageable, consistent, and systematic, we created a template as a guide for extracting data from the program documents. The template included six categories in which data were collected.

Student learning goals

What is it, exactly, that the program wants students to learn? And, is the program clear about what it is attempting to accomplish with regard to student learning? A variety of student learning goals were identified across the university. Examples include effective oral and written communication, analysis and problem solution, critical reading, understanding of the scientific method, and appreciation of the inter-relatedness of multiple disciplines as reflected in historical and contemporary themes.

Learning objective

This category defines what the student must be able to do. The following are examples of learning objectives: the ability to understand relevant concepts and theories, to use statistics to analyze data, and to recognize lines of reasoning and the precise issues they address.

Method

The method is the pedagogical framework, that is, the learning and teaching strategies used for achieving the learning objective. Some programs utilize several non-traditional methods such as case studies, group work, team teaching and simulations, as well as lectures, Socratic questioning, and lab experiments

Measure

This category refers to the assessment tool(s) used to determine whether or not the learning objective was accomplished. Examples of some program measures include traditional exams, writing assignments with grading standards, observed class discussion and participation, oral presentations, individual or group term-end projects, surveys, and individual and group self-evaluations.

Evidence of the accomplished objective

This category indicates to what degree the stated learning objective was attained. The evidence may include demonstrated skills, grades, and survey responses from program participants.

Feedback loop

Here, the template captures how the information gathered in the Measure and Evidence of accomplished objective categories is utilized. How does this information inform the program regarding student learning goals and the articulation of the learning objective? The final category attempts to complete the circle of identifying how the program is focused on student learning, how the program is accountable for student learning, and how the program is committed to continuous improvement to student learning. A copy of the template is included as Figure 1.

Study Questions

In an effort to study each selected department or program in as robust a manner as possible, subcommittee members were encouraged to keep in mind during their reading of program materials or in their interviews with program members the following list of questions. These questions were generated by subcommittee members themselves during the process of refining the six template items described above:

1. What learning outcome(s) is(are) unique to this program?
2. What are the program's goals? For what careers is the program intended to prepare its students?
3. What are the learning standards used to assess student learning?
4. Are students meeting these standards?
5. Is achievement of the program's goals linked to the faculty?
6. Are learning standards linked to employers' feedback and the program's goals?
7. What is the sequence of courses within the program? What is the logic behind the program's prerequisites?
8. Does the program measure student-to-student learning outside of the classroom?
9. Is any information on student-faculty out-of-class learning provided?
10. What is the program's impact on students of various subgroups: gender, frosh/transfer, on/off campus, ethnic breakdown?
11. Is baseline information on student learning included in the report/review?
12. Does the program include a capstone experience?
13. Is there any information on students who have met all of the program's requirements except the senior project? Is the senior project an impediment for students' degree attainment?
14. How are student early departures handled? Exit interviews? Follow-up?
15. Are qualitative comments a part of the review/report?
16. Are case studies included in the review/report?
17. Is there any alumni contact?
18. Does the review/report follow post-graduation activity?
19. If feedback is received from alumni or employers, is it linked with future planning?
20. What measure of progress toward the degree (throughput) is reported?

Top

Findings, Interpretations, and Analysis

During the collection of information for this report some members of our subcommittee found it necessary to promise anonymity to the departments or programs they contacted. These subcommittee members believed that the potential to collect frank information on programs outweighed the potential costs of withholding the specific source of the information from the general campus community. In presenting this report we feel obligated to honor these promises. Consequently, while we may praise certain outstanding practices or criticize others, our comments will not be directly associated with any specific campus program or department.

To avoid overgeneralization, however, we have decided to partition our presentation of the findings in the following broad categories: professionally-oriented academic programs, other academic programs, Cal Poly Plan (PDF) programs, Student Affairs programs, and the General Education program. Our comments in this report remain within the confines of these categories.

University-wide Planning & Review Processes

Curriculum proposal and internal review processes are currently in a state of flux at Cal Poly. Nevertheless, these changes are directed toward incorporating principles that make individual courses and the general programs in which they reside more accountable for student learning. Proposed courses pass through curriculum committee reviews at department, college, and university levels. Before a proposal leaves a department, however, all of its relevant information must be expressed in a standardized format. For example, the 2000 Course Proposal form requests detailed information on a proposed course's learning objectives and criteria. Once these have been identified, the form requests a determination of the specific assessment instruments that will be used to measure whether or not students have attained these objectives. Thus, there is an increasing insistence to focus Cal Poly on student learning.

At a higher level, too, Cal Poly is moving toward orienting its periodic internal program reviews to learning outcomes. Although a final version of the Program Review document has yet to be presented, recent drafts clearly ask programs to consider their activities in the light of student learning goals, measures, and feedback for program improvement. The information currently requested from programs under internal review includes descriptions of educational goals, instructional designs and methods, assessment methods and the data so collected, and the procedures for utilizing the collected information. While not all departments on campus have yet been reviewed with these new guidelines, as time goes by more and more will be asked to adopt this approach to self-examination.

Professional Academic Programs

Our subcommittee studied a sample of eighteen departments or programs that prepare students in specific professional areas. Several departments and programs share objectives which, broadly stated, amount to four main points. First is the intention to develop in students a specific knowledge base with associated analytical and technical skills based in sound theory relevant to their intended professions. Second is the belief students need opportunities to apply their knowledge and skills in field settings. Third is the goal of developing students' social and communication skills. Fourth is the desire for students to acquire the capacity to develop and change professionally as their chosen professions evolve. Four programs mentioned the objective of having students cultivate interdisciplinary skills, and five cited the importance of students learning the ability to fit their skills to diverse social or environmental contexts.

As would be expected, there is a certain amount of variability in the objectives expressed by the programs sampled within this area. This variability is due in part to the breadth of the programs; some objectives came from full departments while others were from smaller programs inherently more restricted in their focus. However, not all of the variability in expressed objectives is explainable this way. It is apparent that programs vary as well in the extent to which faculty have delineated their specific student learning goals. In two cases it seemed that a program's faculty simply had not yet considered for themselves what these goals were. Rather, in trying to articulate its student learning goals, a program appealed exclusively to guidelines provided by an outside assessment or accrediting agency. This stands in contrast to many programs studied in this area in which external guidelines exist, but whose faculty have also expressed customized goals for their students.

Program delivery within this area includes the expected courses offered as part of an integrated curriculum. The professional nature of these departments and programs allows them to utilize experiential education techniques to a great extent. These techniques include internships, field placements, and the Cooperative Education program in which students work in business or educational institutions where knowledge or skills learned in the classroom are actually applied. Another category of student-learning opportunities afforded by these professional programs is the team project. Whether the result of business or industry needs or of faculty research, students in these departments and programs frequently participate in projects and competitions to develop new designs, procedures, or products. Often these projects are conducted within department clubs or classroom laboratories as well.

In determining whether student learning goals have been attained, programs within this area collect a variety of measures. Seven departments in the sample are involved in external reviews or accreditation programs. Their involvement with business or industry allows six of these departments to collect evidence of their effectiveness from industry advisory councils. Feedback about student performance in internships and the cooperative education program is regularly collected as well as data indicating where program graduates are employed. In general, the departments in this area of the university are quite successful at placing their graduates, and many businesses express demand for Cal Poly students in particular. Seven of these programs also collect information directly from their alumni or graduating seniors through surveys, some of which employ pre- and post-test measures to indicate student achievement. Naturally, these departments also use student evaluations of courses, student journals, and student portfolios as indicators of student learning. Finally, a method of assessing student learning mentioned by four programs is the use of a capstone course or project. Again, there is broad diversity in the types of measures these programs collect. Some programs regularly collect large amounts of information from sources outside the university such as businesses or external review agencies. Other programs appear to receive little if any feedback about student learning beyond course evaluations and student grades. As with the identified student learning goals, part of this variability could be attributable to differences in the scope of programs within this category.

Differences among the methods used to measure student learning probably are responsible for differences in the ways that departments and programs use the information they collect. In some cases the information collected from advisory councils is directly supplied to faculty at council/faculty meetings. In other instances advisory boards meet with a department's curriculum committee. Whether or not programs retain an advisory board, many receive input from accreditation agency reports and some keep track of student licensure rates. At the very least, however, most departments report that faculty meet to revise overall curriculum and individual course content on the basis of student feedback. Only two programs reported little if anything in the way of a formal loop in which feedback about student learning is used to improve program delivery.

Other Academic Programs

A variety of issues emerged from the six other academic programs that we sampled. For some departments the articulation of student learning goals represented a paradigm shift. That is, the focus in the university educational setting seems to have shifted to emphasize student learning rather than the traditional focus on teaching. In the process, programs and departments have developed various types of goals. Some focus on a distinct aspect of learning. Some are more general and may draw on a wide range of courses intended, for example, to encourage students "to demonstrate, in other discipline areas, the skills of critical reading and writing." Even though great energy has been spent on the shift in focus to student learning, some programs have found it challenging to state clearly what the program was attempting to accomplish. As a result, some goals appear general or vague.

Across all six of these academic programs, specific methods for learning were well articulated. In fact, the methods included a variety of techniques such as interdisciplinary team teaching, cooperative learning activities, computer mediated activities, field work, and open guided-inquiry labs as well as the traditional lecture. Three of the six programs have identified or partially identified specific learning objectives that are directly linked to their stated student learning goals. The remaining three programs identified specific objectives, but they are not yet directly linked with the student learning goals.

In the area of accountability, several measures were employed among the programs to capture how well they were meeting their stated goals and learning objectives. Surveys appeared to be the most common tool used among five out of six programs currently. Other measures included feedback provided by instructors to homework (including directions for revising), exams, and senior projects. The degree of success in a co-op or internship, success of graduates, and communication with employers are also viewed as useful measures. It seems that what sets programs apart regarding accountability is the degree to which the measures tie directly to the program's learning objectives. Two programs reported such a well designed connection.

The evidence of how well a program met its stated learning goals and objectives varied widely across programs. Surveys of alumni and the outside professional community appear to be conducted irregularly. In one program, grade distributions were claimed to constitute the most basic and accurate evidence of goal attainment. The program devised a method by which grades assigned by various instructors for various courses could all be assumed to reflect core goal-related criteria adequately. Another program found credible evidence in student class presentations, presentations at professional meetings, and paper submissions for peer reviewed journals, as well as a positive report from an external review claiming that the program was outstanding and seemed to have a sense of community.

The degree to which a feedback loop is used to stimulate continuous program improvement varies among programs. For some, changes in the curriculum may occur. However, the changes do not necessarily appear to be driven by a clearly focused assessment process, and information gathered from assessment efforts appears only minimally to inform program goal attainment. One program, however, clearly adjusted course content based on measures of the levels to which students attained their learning objectives. One other program engaged in collegial communication regarding instructional effectiveness for assessing the attainment of learning goals. Two programs had no structured process for feedback at all. One other program reported department meetings, retreats and an Industrial Advisory Board as part of a feedback loop, but no mention was made in relation to specific student learning goals.

Student Affairs

The large scope of the fourteen Student Affairs programs studied for this report may be roughly summarized by their common aim to provide students with the skills they need

• to succeed academically and socially in a university,
• to cope with the stresses of student life, and
• to prepare them for life beyond the university.

Student Affairs endeavors to provide students with as smooth a transition as possible into and out of university life. As a consequence, some of Student Affairs' goals overlap greatly with those of the university's academic programs, yet many others are unique. Some student learning goals likely to be shared with other programs on campus include the acquisition of life-long learning skills, the ability to lead or to collaborate with others, the ability to synthesize information from diverse sources, and the cultivation of a sense of social diversity and justice. On the other hand, Student Affairs aims at easing students into an academic lifestyle they likely have never before encountered, providing students with daily survival skills they may not yet have needed, and offering students academic and career support as required.

No single Student Affairs program can achieve all of the objectives listed above, nor should any one be expected to. Yet, taken as a whole, the programs within Student Affairs have some of the best-defined student learning objectives at Cal Poly. Subcommittee members have described some of the Student Affairs programs as potential models for other programs or departments on campus. To be sure, there were occasional exceptions in which few if any goals were expressed. However, the rule in Student Affairs seems to be one of clearly defined outcomes.

It is difficult to imagine a more diverse set of delivery techniques than in Student Affairs programs, but this is not to detract from them. This potpourri seems appropriately suited to their numerous student learning objectives. These techniques include off-campus work experience and community service programs, mentoring programs, campus-wide fairs and events, as well as more traditional methods such as workshops, seminars, and classroom instruction.

By far the most commonly used measure of program efficacy in Student Affairs is the survey or questionnaire. Twelve of the fourteen programs in this category use this technique to collect information. These surveys ask for self-reported responses from program participants in both quantitative and anecdotal form. In six of these programs some information is collected in the form of pre- and post-test designs, and two programs make use of designs involving the comparison of treatment and control groups. In addition, Student Affairs programs may collect student grades, progress reports from businesses at which students are engaged in work experience, information from off-campus visitations of these work sites, or student evaluations of their program experiences as appropriate. With only spotty dissension, there is general satisfaction that these programs are meeting their intended goals. Many of the reported student anecdotes and evaluations bear this out.

Feedback in Student Affairs is often compiled in the form of reports. Whenever appropriate, student feedback is also given directly to instructors, counselors, or discussion leaders. This may be in the form of either statistical summaries of student responses or transcripts of open-ended student comments. Eleven of the fourteen programs reported the regular collection of feedback. In a few areas within Student Affairs this feedback seems to serve only as a confirmation that a program is serving its intended function. Beyond this basic role, however, some programs expect the collected information to provide an indication of potential improvements in program delivery and program self-monitoring. Thus, some Student Affairs programs use feedback not only to change their program's method or delivery, but also to change the program's assessment techniques themselves (such as the composition of a survey, for example). Improvement of assessment procedures-tailoring the information collected to report more effectively on a program's intended goals-is a hallmark of many Student Affairs programs.

Some Student Affairs programs do not use feedback as a regular part of their activities. Occasionally it was unclear how assessment data were being used to improve the program, let alone improve the program's assessment efforts. At other times it was not clear which specific improvements had been made on the basis of feedback. Some subcommittee members felt that no information was available about how instructors used feedback provided through Student Affairs programs to improve their courses. Others felt that instructors could provide more information than they currently do by comparing the academic preparedness of students who had previously participated in Student Affairs programs with students who had not participated.

Another concern focused on the inability to determine whether it was, indeed, the program that produced a measured effect in students' abilities to attain learning goals. Specifically, in a program that utilized a regularly offered course, it seemed as if students taking the course but not participating in the Student Affairs program achieved results similar to those of students in the course who were participating in the program. The question then is, did the program produce any effect above and beyond the course itself? This is another problem regarding study design and data analysis. In one anomalous case the question was raised of whether a single survey was sufficient to measure all of the intended outcomes of a particularly ambitious program.

Student Affairs has integrated assessment into the ongoing operation and periodic review of most of its programs. Many programs within the division have defined learning outcomes and developed questionnaires to sample opinions about program efficacy. As in the rest of the university, not all programs have engineered feedback loops. Consequently, what is learned about program effectiveness is used by some but not all programs.

Additionally, the subcommittee has identified two problems regarding data collection and statistical inference. First, as we have implied above, the analysis of what affects learning is not always precise. For instance, reporting instruments do not always clarify whether learning is due to student participation in a whole program, the separable courses within it, or external variables. Second, the assessment procedures, the techniques for reporting data, and the conclusions drawn from them often do not follow accepted statistical practice. For example, committee members found shortcomings in the choice of sampling strategies and subsequent analyses, the relation of reporting scales to the data collected, and the measures of variability (standard error, confidence intervals, etc.) that put into context patterns found in the data. Indeed, it could well be that proper analysis would confirm the effectiveness of these programs; however, current practices make it difficult to substantiate such conclusions statistically.

The committee's comments are of a technical and institutional nature. Assessment of all kinds requires collaborative design and implementation calling on diverse expertise, because it is complex and decentralized throughout the university. Assessment, therefore, challenges the campus to develop further a conversation in which the following occurs:

1. The people who staff programs dialogue with assessment experts in order to gain more insight into the methods of rigorous measurement and assessment;
2. The people who are expert in assessment and statistical practices learn more about the intricacies of programs across campus so that they can provide theoretical as well as practical advice.

To this end, the Provost recently established the campus Task Force on Institutional Accountability and Assessment.

Cal Poly Plan Programs

The Cal Poly Plan (PDF) is designed to promote institutional productivity, student learning and progress, educational quality, and accountability and assessment in a time of growing enrollments, high public expectations, and limited public resources. The Plan seeks to generate academic fees to support programs that provide a demonstrable improvement in student progress and educational quality. Program proposals are submitted to a Cal Poly Plan committee and evaluated for funding. Previously funded programs represent several different categories relating to student progress to degree and educational quality; specifically they focus on institutional productivity, academic enhancement and experimental efforts, and innovative teaching and learning.

The subcommittee reviewed nine programs funded by the Cal Poly Plan. Two of the nine were for one-time investment purchases of hardware to benefit student progress and educational quality by providing access to state-of-the-art equipment. These programs had clearly defined student learning goals. Learning objectives centered on acquisition of conceptual, behavioral, and procedural skills. Methods for achieving the stated learning objectives included workshops, classes, and student projects. Surveys, self-evaluation, and videotapes of student accomplishments were viewed as measures of achievement of student learning goals and objectives.

Three out of nine funded projects categorized under Institutional Productivity as "academic enhancement and experiential programs" had very distinct and clearly defined student learning goals, learning objectives, and instructional methods. The assessment instruments utilized for these programs involved surveys administered at various times, analysis papers, mentor evaluations, and evaluations from external, non-academic organizations. There appears to be a general acceptance that these programs are in fact meeting their stated goals. Program participants appear to have grades equal to, or better than, non-participants, indicating that their success in improving cognitive skills has been at least as good as that of non-participants. At this time, it is not yet clear whether or not the program itself was responsible for the successes. Furthermore, there appear to be few examples of specific and major improvements in the program based on particular assessment evidence.

The last three Cal Poly Plan funded programs that were reviewed were characterized as using "innovative teaching and learning" that contributed to student progress to degree. These three programs, collectively, had well defined student learning goals. One program, in particular, accomplished its student learning objectives through the establishment of a learning environment that evolved as the learners grew. Technology-based and interactive and collaborative web projects were other methods used to accomplish the learning objectives. The forms of measures used for assessment included weekly student journals, instructor journals, participant evaluations, and technology based feedback. Evidence of the program's success in accomplishing the stated student learning goals was demonstrated by the number of students and faculty from the program who were recognized for their work.

One program within this category had very specific learning objectives that were tied to the overall college learning goals. The program developed innovative methods for addressing specific learning objectives such as multimedia presentations, simulations, case studies, long-term group/team projects, and integrated team teaching. As was the case with many other programs, exams, oral presentations, writing assignments, participant self-assessment, and surveys linked directly to the learning objectives were utilized for overall program assessment. The evidence of how well the student learning goals were attained included oral feedback from students, successful completion of the course, grades, and survey responses. In one case, the design included a comparison group that was drawn from a course with similar types of students as the participants, but the data analysis did not meet accepted standards.

Because the program is fairly new, the analysis was informal and no strong conclusions were reached. However, the program is committed to a comprehensive study of its impact in the near future. The fact that industry actively seeks student participants from this program is also viewed as evidence of the strengths of the program. Though this is certainly a strong complement to the program, it may not be linked to specific learning objectives and, therefore, may not be direct evidence that they have been achieved. All Cal Poly Plan funded programs have been subject to a review of their intended learning outcomes, both at the proposal stage and after each year of operation. Therefore, it was evident that these programs were by design most clearly focused on the accountability for student learning and the commitment to purposeful and continuous improvement in student learning.

General Education

Cal Poly believes that General Education (GE) is central and vital to each student's university experience. After reviewing the GE Curriculum which had been in place for the past fifteen years, the Academic Senate spent two years developing recommendations for a revised program to better prepare Cal Poly students for the challenges of life-long learning and effective, engaged citizenry in the twenty-first century. Following the recommendations of the ad-hoc General Education Committee, the Senate forwarded its recommendations to the President who, in April 1997, approved the revised General Education template and program structure for the implementation of GE 2001.

The template calls for 4-unit courses in order to offer fewer courses with greater coherence. During the 1997/1998 academic year, the GE committee and the three Area Committees developed the standards for the new program. Great energy has been devoted to identifying program goals and learning objectives for each of the five areas: Area A (Communication), Area B (Science and Mathematics), Area C (Arts and Humanities), Area D/E (Society and the Individual) and Area F (Technology). In Fall 1998, faculty prepared GE course proposals adhering to the respective area goals, learning objectives, and methods of assessment for Fall 2001 implementation.

The program structure is comprised of a program director and a standing GE Committee charged with monitoring and assessing the GE Program. The GE director and committee are now in the process of deciding what and how to monitor and to assess all the various aspects of the program. The process has involved consultation with other universities, and the assessment plans are scheduled to be formulated by the end of the 1999/2000 academic year.

Top

Discussion, Recommendations, and Conclusions

Best Practices

The subcommittee would like to recognize the following departments or programs for their outstanding commitment to student learning outcomes. Because this identification violates our general policy of anonymity, we requested and received from these programs permission to mention them by name. They are listed in alphabetical order.

Cal Poly Plan Programs

By design, the Cal Poly Plan (PDF) (CPP) programs represent a "best practice" because of the clear focus on student learning. The three goals of the Cal Poly Plan include:

1. educational quality,
2. student learning and progress, and
3. institutional productivity.

All contribute to the end of improving the quality, effectiveness, efficiency, and accessibility of higher education. Through the Plan, Cal Poly seeks ways to decrease student time to degree, increase student learning, enhance institutional productivity and productivity in teaching and learning, promote the more effective use of fixed resources, and implement comprehensive assessment and accountability procedures. The Plan will support new ways of educating and supporting students, including creative approaches to teaching and learning and their measurement, curriculum design and scheduling, and the application of information technology to instruction. These efforts require multi-year investments in human resources, mainly in the way of professional development for faculty and staff, as well as in equipment.

The CPP request for proposals required that the content of any program be focused on student learning, accountable for student learning with a specific plan for program assessment, and committed to purposeful and continuous improvement in student learning. Specifically, the proposal required very clear program goals with strong learning objectives that directly address the goals of the program. Also built into the design of CPP funded programs was the requirement to report yearly the program accomplishments in achieving the stated program goals and/or plans for improving the attainment of goals in the future. CPP funded programs were reviewed by a university-wide committee comprised of members from the Cal Poly Plan Steering committee representing all campus constituents: students, staff, faculty and administration.

College of Business' Undergraduate Integrated Core Program

The College of Business' undergraduate Integrated Core Program (ICP) clearly-states program goals and learning outcomes which are tied to the college's overall learning goals developed by college committees of the COB Advisory council. The program makes use of several progressive-teaching methods such as active learning, group work, simulations, case studies, and team teaching to achieve its learning outcomes. Student participants reported favorably in the following areas: student-faculty relations, faculty accessibility, enthusiasm and dedication, multiple faculty members involved, camaraderie and climate of the class, group experiences and the capstone assignment of a full-scale business plan.

The ICP gathers assessment data at various times during the course of the program. One of the instruments, the core evaluation survey, directly reflects the program's learning outcomes, and data are collected from a control group of non-participating business students to compare with participating students' survey scores. Feedback is also collected from an industry advisory group. All of this information is used by the college curriculum committee and by a faculty team to improve course delivery.

Cooperative Education Program

This program aspires to several clearly expressed and highly focused student learning goals. While the nature of the Co-op program prescribes work placement as its lone educational method, a variety of measures are taken to assess whether the program goals are being achieved. These measures include Career Services interviews with students about their Co-op experiences, and the use of student pre- and post-test surveys. Career Services also conducts periodic site visitations of the participating employers. Together, these measures provide efficient and converging evidence of the program's effectiveness.

It is very clear that Co-op has become a successful and integral part of many academic programs on campus. The widespread and growing utilization of this program is reflected in the positive feedback provided by students, faculty, and employers. This success should come as no surprise given the tremendous efforts of the Career Services personnel in making Co-op the unique contribution to a Cal Poly education that it is. Following the recommendations of the WASC report from a decade ago, Career Services has made the collection and use of feedback a regular and integral feature of the Co-op process. That they take to heart the information they collect is made obvious by their constant striving to refine and to improve their data collection techniques. In terms of the energy and dedication they bring to improving their program, they are nonpareil.

Psychology & Human Development Department

All of the goals of the Psychology & Human Development department are focused on aspects of learning. The goals capture the broad domain of student learning in the department's two major areas and are sufficiently unique in differentiating this department from similar departments at Cal Poly. Their learning objectives are for the most part concrete and measurable, and they place this program far ahead of others in the development and use of learning objectives. Psychology & Human Development uses varied and appropriate methods for the attainment of their goals. Grades are tied directly to learning objectives in the majority of the courses taught, and other measurements include students' self-reported level of learning or improvement. The limited nature of such self-reporting is acknowledged by the program, but all questions tie directly to learning outcomes.

Individual faculty use feedback to adjust course content according to the reported attainment of objectives. The Program is also considering the establishment of a standing committee to evaluate senior projects and internship reports to determine whether learning objectives have been attained in those areas. The committee would feed these results back to the program for course content or curricular redesign. This department is highly focused on student learning, and the only recommendation for Psychology & Human Development is, in the words of the subcommittee members who investigated the department, to "keep on keeping on."

The effect of external program review

Some of the departments and programs sampled for this report participate in periodic external program reviews or accreditation processes. The question of what influence these additional activities may have on the assessment of student learning was discussed among subcommittee members. Although no formal, quantitative approach was taken in examining the relationship between external review and student learning, subcommittee members came to consensus on two conclusions. First, it is apparent that participation in external review provides a program with more information about its efforts than might otherwise be collected. This does not, however, guarantee that the information is effectively linked with either the program's self-identified learning objectives or the objectives identified by the external agency. Second, even if there is a link between the information collected and the program's learning objectives, external review does not necessarily imply that this information is utilized in improving the program's goals or delivery. Expressed another way, the "feedback loop" is not necessarily closed simply because feedback was collected at the request of an external agency. It was the impression of the subcommittee that both strong and weak assessment practices could be found both in programs that did and did not participate in external review.

Can a single assessment model fit all departments?

During the course of our subcommittee's work we debated the issue of whether or not a single assessment template could adequately apply to each and every department or program. We would like to make it clear that we did not view our charge to include the imposition of any specific style of student learning assessment on the campus community. In no way was it our intention to impose, for example, a single set of learning goals or objectives on any department or program. From the start this subcommittee recognized that any given program would have unique objectives that only its faculty could delineate. As our work progressed, it became obvious that some departments had not considered overtly what those goals might be, but we cannot necessarily take them to task for that. This was not to be an assessment of how a program's self-reflection efforts matched our particular approach to studying student learning outcomes at Cal Poly. It was not an effort to insist on universal learning objectives or outcome measures across that can be applied to all disciplines. It was an effort to bring a modicum of standardization to the study of student learning outcomes so that apples and oranges could be discussed in the same document.

Our subcommittee does not advocate any particular variety of learning outcome assessment. We do advocate the strong correspondence of learning outcome measures, whether quantitative or qualitative, to clearly expressed relevant learning objectives. However, individual departments and disciplines must be allowed to establish their own legitimate objectives and the assessment strategies that will be used to measure them based on the legitimate complexities of their programs. Some programs on campus happen to be farther along than others in this process. It seems clear that all departments and programs at Cal Poly will soon be involved in this effort, and each should be allowed to retain its individual academic character in the process. Some subcommittee members expressed their concern that the goal of expedience in the assessment of student learning might blind those involved in such assessment to the complexities of various academic endeavors. We could not agree more with the spirit of this concern.

Recommendations

Closing the feedback loop

As a result of examining a wide variety of program documents and interviewing program coordinators, chairs, and directors, there appears to be a consistent failure to use what our Subcommittee refers to as the "feedback loop." This refers to how the assessment evidence informs the program for improvement in order to enhance attainment of the stated learning goals and objectives. Two key patterns emerged among the programs studied regarding this important aspect of the learning outcomes approach. The first is that many assessment efforts did not directly link with a program's stated goals and objectives, thus precluding the possibility of generating useful information for focused program improvement. Second, feedback loops were often characterized as collegial conversations, department meeting discussions, or presentations to advisory boards regarding particular programs. However, no mention was made of the discourse being purposefully and directly focused on the program's specific goals and objectives. The subcommittee recommends periodic reviews of the feedback loop(s) to maintain program effectiveness in student learning outcomes.

These shortcomings may be attributable to the fact that not all programs are aware of, committed to, or mandated to operate in a focused learning outcomes mode. Or perhaps the programs are, in fact, operating within such a mode but simply not documenting their efforts as such. Regardless, there appears to be inconsistency in approaching this issue across campus. If, however, the university is committed to the learning outcomes and assessment approach, a campus-wide mandate with clear expectations may be warranted. Such a mandate must, however, recognize that some programs and disciplines may recognize a natural fit to such an approach, while others may challenge the model and arrive at a modified, more discipline-specific approach.

Improvement of research design and data analysis

This recommendation incorporates all areas of assessment. It includes the initial choice of a data-collection design, the relevance of the data collected to the question(s) under study, the techniques used to gather the data, the techniques used to analyze the data, and the conclusions about student learning that may or may not be properly drawn from this process. The process must begin with a clear expression of what is to be assessed and an understanding of the practical constraints of collecting data in field settings, settings that do not permit the use of optimal designs for establishing cause-and-effect relationships. Despite these inherent limitations, designs must be selected that can strengthen the claim that a program is producing an imputed student learning effect, should one indeed exist.

Beyond the careful determination of the assessment goal and design, more consideration must be given to the type of data collected. We have found that surveys are the dominant form of data collection on campus, and while this is understandable, we encourage the cultivation of other techniques as appropriate. Wherever possible, multiple measures should be used in order to provide converging evidence of a program's effectiveness. In any event, the instruments used to measure the attainment of an objective should be specifically designed for that objective. The blurring or blending of objectives within a single measurement instrument often provides useless information. However, their construction is not to be taken lightly. It is critical that survey content be clear and directly linked to a program's explicit student learning goals. Despite a survey's deceptive simplicity, these aims are more easily stated than achieved.

More attention needs to be paid in the assessment process to the nature of the populations under study. Often no clarification of a target group is made, or several distinct groups may be of simultaneous interest, e.g., current students, recent alumni, or industry/employers. These commonly targeted groups aside, we feel that populations underrepresented in a particular program or special populations campus-wide are not often enough the target of these assessment efforts. Greater pains should be taken to measure program impact on these groups in order to better attract and accommodate them in the future.

Once data have been collected, they must be analyzed in an appropriate manner. This includes considering the nature of the measurement scales used, determining whether the assumptions of a statistical procedure have been met, and using inferential statistics to draw generalized conclusions from the samples which have been collected. The proper interpretation of what research data say and do not say about student learning rests squarely on rigorous analysis. Indeed, we believe that the ability to make justified, well-supported claims about student learning at Cal Poly will require following every one of these design and analysis guidelines.

Linkage of student learning objectives with program goals

All campus programs must clearly define their goals. In addition to this, however, the specific learning objectives that programs establish for their students and later attempt to measure via the assessment process must be directly linked to the program goals. This linkage is critical for measuring attainment of the goals.

---

Figure 1: WASC Student Learning Committee - Information Extraction Template

I. Student Learning Goal	II. Learning Objective	III. Method	IV. Measure	V. Evidence of accomplished objective	VI. Feedback loop

Guidelines for Senior Projects (from CAM 412)

412.1 Definition

The Senior Project at the California Polytechnic State University, San Luis Obispo is a formal report of the results of a study or experiment selected and completed under faculty supervision by each student prior to the receipt of the bachelor's degree. The types of problems which form the bases of the study or experiment are directly related to the student's fields of employment or intended employment.

412.2 Expected Outcomes

1. Ability to reduce a general problem to specific points of analysis
2. Ability to organize points of analysis into a logical sequence
3. Ability to estimate hours of labor and cost of materials necessary to complete a project
4. Ability to apply competencies acquired in other courses to the successful completion of a specific project
5. Ability to obtain information necessary to the solution of a problem by library study, experimentation, and/or correspondence and personal contact with people who have had experience in the field
6. Ability to follow a work outline without overlooking any major points or significant details
7. Recognition of the fact that completion of a project on schedule is an essential element of successful work
8. Ability to organize, illustrate, and write a clear, concise, and correct report of the investigation
9. Ability to work for a supervisor who desires quality performance with a minimum of supervision

412.3 Requirements

1. Every student must complete satisfactorily the Senior Project prior to the receipt of the bachelor's degree.
2. The number of quarter units of credit for Senior Projects must be within the range of 2 to 4.
3. The specific number of units required would be the same for all students in a given curriculum, but not for all students in the university, because of the nature of the various curricula.
4. A minimum of 30 hours of student work will be required for each unit of credit granted.
5. The character of each curriculum will determine the method of organization of the course requirement, i.e., lecture or activity.
6. One or two quarters of work may be specified for the various curricula depending upon the nature of the curricula.
7. The responsibility for costs for materials and supplies used in the project will be determined in advance by the university. Costs should be borne by the student when the product of the project is for personal use.
8. The number of students involved in any given project should not be so large as to limit individual experience or responsibility and initiative. Each student should be required to meet meaningfully the 30 hours per unit of credit minimum.

412.4 Library Copy

1. One copy of each Senior Project will be sent by the academic department to the University Library where it will be copied on microfiche. A microfiche copy of the project will become part of the Library's collection where it will be available for public use. One copy of each microfiche project will also be deposited in the University archives.
2. Each student is required to pay a fee for copying his/her Senior Project on microfiche.
3. After being copied on microfiche, the original project will be returned to the academic department of its origin. Non-print media (slides, audio/video tapes), however, comprising all or part of a project will be permanently retained in the Library collection.
4. All projects submitted to the Library will follow a standardized format for title page, approval page, and abstract. Details of this format are found in Procedures for Submitting Senior Projects to the Library, available from the University Archives in the Library.