Surry Community College - Distance Education

Objective 2: Assess Learning Outcomes

Narrative

The College will agree on learning outcomes relevant for all degree programs, on strategies to improve learning outcomes, on assessment processes to measure the acquisition of learning outcomes, and on means for documenting achievement of outcomes.

It is well-documented that improving cohesion in the curriculum enhances student learning. General education outcomes are not the responsibility of particular divisions. Basic assumptions underlie the development of SCC’s assessment program for general education. These assumptions are as follows:

Learning outcomes assessment is a natural and ongoing component of the instructional process.
The process of assessing learning outcomes is a means to an end, that end being improved learning.
In no instance will the results of learning outcomes assessment be used in a punitive manner, either in reference to students or to personnel.
The faculty, given their curricular role and responsibility, will have primary responsibility for the development, implementation, and maintenance of assessment activities.
Faculty must explicitly link course objectives to general education goals.
Attainment of General Education goals is achieved within the context of a degree program.

Assessment Model Adopted

After an exhaustive literature review in the fall of 2002, the Council for Innovation and Student Learning adopted Johnson County Community College’s (JCCC) model for assessing General Education outcomes. The core features of the JCCC model include 1) integrating performance-based assessment projects into classes that ask students to apply what they know and to integrate knowledge and skills in complex performances; 2) using the course/class as the unit of analysis; 3) embedding approaches that make use of student work samples and performances gathered through regular coursework. An attractive feature of the JCCC model is that assessment is not an "add-on"; rather, classroom projects are utilized to provide ample evidence of student learning and success that evolves from and can be considered in an authentic context. At subsequent meetings throughout the fall, CISL decided that adoption of the JCCC assessment model (the Institutional Portfolio) could provide an opportunity for students to engage in meaningful class projects. Such projects require students to integrate and apply knowledge and skills while simultaneously providing assessment data for General Education outcomes.

Overview of the Institutional Portfolio

The Institutional Portfolio model requires the collection and review of student projects produced in courses throughout the curriculum for each of the General Education learning outcomes and the development of rubrics for assessing outcomes at an institutional level. It is based on models developed at Johnson County Community College, Overland Park, Kansas (See: Jeffrey A. Seybert and Kathleen A. O’Hara, "Development of a Performance-Based Model for Assessment of General Education," Assessment Update, Vol. 9, July-August 1997) and at Shawnee State University (See: materials from workshop conducted by David Todd, "Assessing Writing and Critical Thinking Skills Using a Rubric Applied to Portfolio Entries," AAHE Assessment and Quality Conference, Cincinnati, Ohio, June 1998).

In this model, the review of student projects (artifacts) is conducted by interdisciplinary faculty teams using holistic scoring criteria (rubrics). Assessment results are reported for the College as a whole but may also be disaggregated and analyzed by a number of demographic variables of interest to the faculty (for example, credit hours earned or prior courses completed).

Artifacts are randomly selected by the Institutional Research Office: One hundred twenty artifacts will be collected for each of the SCC learning outcomes. Faculty of the randomly selected classes will be asked midway through the semester to collect assignments. Faculty will submit the artifacts to the Office of Institutional Research, which will remove any information that identifies either the student or faculty associated with the artifacts and make additional copies of the artifacts for distribution to the faculty scoring teams. The Office of Institutional Research will retain copies of the original, identifiable artifacts for follow-up work to disaggregate the data. These artifacts will be maintained in a secure, locked environment in accordance with the requirements of the Family Educational Rights and Privacy Act. The Office of Institutional Research will forward copies of the artifacts—minus identification of student, faculty, and class—to the appropriate faculty scoring teams.

Faculty scoring teams will assess the artifacts in terms of Primary Trait Analysis or rubrics. The scored artifacts will be returned to the Institutional Research office, where the data will be archived. By the end of the academic year, the data will be analyzed and reported to the Council for Innovation and Student Learning, the Learning Outcomes Team, and the faculty at-large.

The collection, review, and summary of student work produced throughout the curriculum for each major outcome will comprise an Institutional Portfolio. The following list details the logistics of the Institutional Portfolio.

120 artifacts collected per learning outcome per year
Office of Institutional Research selects courses
Faculty in each targeted class select artifacts
Three-to-four-person interdisciplinary faculty teams score per outcome using holistic scoring criteria (rubrics). Student work is scored individually by team members or as a group (one score reported per artifact)
Office of Institutional Research collects, copies, distributes artifacts
Results compiled by Office of Institutional and reviewed by faculty who propose curricular improvements
Budget $8,000 annually for incentive grants for faculty, stipends for scorers, plus significant support from Office of Institutional Research
Annual review by Learning Outcomes Team for assessment of assessment plan

The following list summarizes the main characteristics of the Institutional Portfolio model

The outcomes and scoring teams are multidisciplinary; thus responsibility rests with the institution/faculty as a whole, rather than with single departments.
It is invisible to students, obviating the motivation and other significant problems with standardized tests.
It is minimally intrusive for faculty.
It requires no special "sessions," no sacrifice of class time (e.g., for testing), no external incentives for students to perform well.
It is labor-intensive and requires significant institutional resources (faculty release time and/or overload pay, technical support).
It is a dynamic process.

Learning Outcomes Identified

CCSSE revealed that faculty and staff felt that (a) students need greater involvement in higher-order thinking skills and (b) the college needed to develop a more adequate assessment process of general education learning outcomes. Dr. Steve Atkins and Dr. Susan Worth, LOT co-chairs, surveyed faculty and staff in the fall of 2002 to determine learning outcomes relevant for all degree programs. Dr. Atkins and Dr. Worth presented the survey results to CISL, and the Council identified and defined five learning outcomes (i.e., communication, quantitative literacy, critical thinking, culture and ethics, and information and technology skills) relevant for all degree programs. CISL developed and adopted a working draft of each of these core learning outcomes (described below).

CISL decided that critical thinking should become the “umbrella” outcome: The other learning outcomes identified (communication, information literacy and technology skills, quantitative literacy, and culture and ethics) are specific ways in which critical thinking can be fostered, practiced, and evaluated. Further, critical thinking has been consistently mentioned as an area of concern in every faculty focus group, survey, and analysis completed at the College in the last three years and thus seemed a reasonable and effective focus for improving student engagement campus-wide. Critical thinking teaches students how to do the intellectual work required of the other outcomes, it promotes active learning strategies and authentic assessment, and it is easily applied across all disciplines and programs. Although critical thinking is only one of SCC’s learning outcomes, it has become the major focus for improving student engagement across all disciplines.

The Council developed the following definitions for each outcome:

Critical Thinking: Students who graduate from Surry Community College should be able to demonstrate skill in “conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication ” (Paul & Scriven).

Communication:

Communicate effectively through writing: Students who graduate from Surry Community College should be able to produce writing that is clear, precise, organized, incisive, and correct (according to the guidelines of Standard Written English) for a variety of purposes and audiences.

Communicate effectively through speaking: Students who graduate from Surry Community College should be able to speak in a manner that is clear, precise, coherent, perceptive, audience-aware, and correct (according to the guidelines of Standard English) in both small and large group settings.

Communicate effectively through reading: Students who graduate from Surry Community College should be able to read actively and analytically at the college level and should be able to synthesize and apply information across disciplines.

Information Literacy and Technology Skills: Information Literacy: Students who graduate from Surry Community College should be able to recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information.

Technology Skills: Students who graduate from Surry Community College should be able to use computers, software applications, databases, and other technologies to achieve a wide variety of academic, work-related, and personal goals.

Quantitative Literacy: Students who graduate from Surry Community College should be able to apply college-level mathematical concepts and methods to understand, analyze, and communicate in quantitative terms.

Culture and Ethics: Students who graduate from Surry Community College should be able to demonstrate an understanding of social, professional, or environmental issues that will enhance their capacity for making sound ethical judgments as well as demonstrate a respect and understanding of diverse people, including those with different cultural and linguistic backgrounds or different physical abilities.

The communication learning outcome was divided into three sub-skills (reading, speaking, and writing) forming three committees. The information literacy and technology skills learning outcome was separated into two committees. These five, together with committees formed from the other three learning outcomes, formed the eight sub-committees of the Learning Outcomes Team. Eight faculty members, chosen by Dr. Atkins and Dr. Worth for noted expertise in the particular area, assumed the role of chair for each committee.

The Learning Outcomes Team (co-chairs and sub-committee chairs) met on January 21, 2003, to review the SCC Learning Initiative. Sub-committee chairs were asked to select a diverse group of faculty and staff to serve on each committee. It was decided that chairs would specify standards indicating development of the core skills at various levels and that each LOT sub-committee would create a performance-based assessment rubric.

It was also decided that members of the Learning Outcomes Team would lead workshops to aid the faculty in the use of the rubrics and serve as assessors of students’ performance-based projects. Each program area would identify the core skills taught, report to the Vice-President for Instruction with this information, and build the assessment of these core skills into course outlines.

At a May 5, 2003, LOT meeting, Dr. Atkins and Dr. Worth shared resource material on rubric development and assessment, and sub-committee chairs reported progress on rubric development. Dr. Atkins and Dr. Worth suggested that each sub-committee attend the critical thinking workshops led by Dr. Linda Elder and then reconvene to integrate critical thinking concepts into their rubrics. Rough drafts of all rubrics were called for by July 1, 2003, though only the Critical Thinking, Writing, and Speaking outcomes were selected for evaluation in fall 2003.

On July 23, 2003, the following teams met to pilot their rubrics with student artifacts: Critical Thinking, Writing, Reading, and Speaking. Co-chair Susan Worth analyzed data from this piloting exercise to determine inter-relater reliability, to assure internal consistency and fairness in the final ratings, and to refine the rubrics. In September, 2003, sub-committee chairs for Critical Thinking, Writing, Reading and Speaking met with Dr. Atkins and Dr. Worth to further refine and align the rubrics by establishing a 4-point scale using consistent performance level terms: “Exemplary,” “Satisfactory,” “Below Satisfactory,” and “Unsatisfactory.” Additionally, each chair created a worksheet to facilitate use of the rubrics.

Further piloting sessions were conducted in October and November, 2003, for the Critical Thinking and Speaking rubrics. Quantitative Literacy and Reading were selected for evaluation in the spring of 2004.

TOP

Table Summary

Expected Outcomes	SCC Progress to Date
The College will conduct thorough assessments of current programs to determine 1) the extent to which learning outcomes are currently in use and 2) the effectiveness of such programs.	Determining the extent to which learning outcomes are in use: Summer, 2003: Vice-President for Instruction asks faculty to identify learning outcomes addressed in each course, creating a “Learning Outcomes Matrix.” These are updated in fall, 2003, after faculty receive training in SCC's learning outcomes. Determining the effectiveness of such programs: August, 2003: Faculty meet in divisions to discuss how they can improve existing assignments to foster critical thinking and to share ideas for more effectively integrating critical thinking strategies into daily activities.
The College will design a framework for learning outcomes for the institution.	September, 2002: Faculty and staff are surveyed to determine learning outcomes relevant for all degree programs. October, 2002: CISL analyzes outcomes survey results. November, 2002: CISL adopts five learning outcomes and developed working definitions for each outcome. April, 2003: Learning outcomes are approved by Academic Council.
The College will agree on a variety of appropriate assessment processes the faculty can use to measure learning achieved.	January-December, 2003: Rubrics are created for Critical Thinking, Writing, Speaking, Reading, and Quantitative Literacy. August 12, 2003: Faculty are introduced to Critical Thinking, Writing, and Speaking rubrics; adaptation of the rubrics for classroom use is encouraged but not required. February, 2004: Completed Writing, Speaking, and Reading rubrics are published in the QEP newsletter. March, 2004: Critical Thinking rubric published in QEP newsletter.
The College will apply an outcomes-based process for assessing general education outcomes.	November, 2002: Council adopts Johnson County Community College’s model for assessing general education outcomes. August 12, 2003: Meeting held with SCC faculty to present and pilot rubric drafts for Critical Thinking, Writing, and Speaking. September 3, 2003: Meeting with Critical Thinking, Reading, Writing, and Speaking chairs to discuss refining rubrics. Fall Semester, 2003: Actual piloting of Institutional Portfolio for Critical Thinking, Writing, and Speaking outcomes. October through December, 2003: Additional piloting sessions are held for Critical Thinking and Speaking rubrics, initial piloting session held for Quantitative Literacy. Spring Semester, 2004: Reading is added to Institutional Portfolio.
The College will identify means for explaining SCC’s education objectives to entering students.	April, 2003: CISL develops means for explaining SCC’s education outcomes to students via the Catalog and ACA textbook. April, 2003: Academic Council approves catalog additions. November, 2003: Learning outcomes posters which explain SCC’s education objectives are placed in every classroom on campus.

TOP