Research on Masteries and Assessment: Old Dominion Summary Return to main summary of Research on Masteries and Assessment.
INTERVIEWS WITH SANDRA WATERS, DIRECTOR OF UNDERGRADUATE PROGRAMS, OFFICE OF ACADEMIC AFFAIRS; AND WORTH PICKERING, DIRECTOR OF ASSESSMENT AND WRITING CENTER 1. How did you determine the competencies and who decided? In 1998, the State Council of Higher Education for Virginia (SCHEV) mandated 6 core competencies as part of a statewide general education program &endash; (1) writing, (2) information technology literacy, (3) quantitative reasoning, (4) scientific reasoning, (5) critical thinking, and (6) oral communications &endash; for implementation in all public two-year and four-year institutions. The first Annual Reports of Institutional Effectiveness (ROIE) was released in July 2001, although the first ROIE (aka competencies assessment report) for the first two competencies (writing and technology literacy) was released in July 2002. Writing and technology/information literacy were to be fully implemented by the year end 2003, and mathematical/quantitative reasoning and scientific reasoning are to be implemented beginning 2004 and completed by year end 2005. Oral communication and critical thinking begins 2006 for completion year end 2007. Annual ROIE's are collected and released reflecting progress. ROIE's are divided into 4 sections &endash; 1) institutional mission, 2) system wide measures, 3) institution-specific measures, 4) core competencies. See www.schev.edu for more information. 2. How are the competencies assessed? What, if any, impact do these assessments have on graduation? SCHEV has guidelines for institutional plans to assess and report competencies. These include a framework for reporting competency assessment results. An example reflects a definition or writing competency, standards for competency, description of assessment process, and summary of results. The actual definition, standard, and process are left to the individual institutions. Only two competencies have been completed for reporting purposes, writing and technology literacy. Writing assessment stands alone, since it has been an assessed competency since 1980. Writing assessment consists of the student passing the Exit Examination of Writing Proficiency (EEWP). The student must write an essay that reflects critical thinking on a complex topic and a maturity of expression, including a varied syntax and vocabulary appropriate to the task. The essay must be reasonably free from mechanical errors. The essay must include the following components: 1. a clearly stated objective, 2. ability to present supporting evidence, 3. clarity in sentence structure and word choice, 4. logical consistency, and 5. reasonable freedom from mechanical errors. See attached for more detail on the writing assessment scoring and methodology. Technology literacy and the other 4 competencies are and will be imbedded into courses. The assessment of the Technology/Information literacy competency is broken down into 5 competency sections &endash; 1. systems hardware, 2. system software, 3. application software, 4. telecommunications, and 5. computers and society. The student proficiency in the overall competency by a "pass" grade on each of the 5 sections of the Computer and Technological Literacy Examination (CTLE), a newly designed common examination administered to all students enrolled in three lower-division courses (business, education, and computer science). Instructors are given an approved spreadsheet with the required components for the sections, and the instructor designs the course and the test and does the assessment as far as "pass." The course and measures have to be approved and are reviewed (regularly but not less than every two years) by a Computer Literacy Standards Committee. 3. Is the process based on completing a set of courses or is it more product driven? Both. In Writing, each student takes 2 semesters of composition courses as a freshman. Students may take an entry test and place out of the composition courses with credit. Then they have a writing intensive course as part of their major discipline as a junior or senior. The instructors determine the products in each course, essays, reports, research papers, etc. and the assessment criteria and grade. All courses in the university are supposed to have a writing component, but the composition core courses and the writing through the discipline course are counted toward the competency. Then the student must take the EEWP. If it is failed, the student may not graduate and must go to a writing workshop at the Writing Center. In Computer Literacy, a spreadsheet of 20 required competency components has been developed and given to the instructors in 3 competency courses. There is one in business, one in education, and one in computer science. A student must take one of the courses. With help from the Assessment Center, the instructor develops a mid-term and a final test which incorporate the 20 components. The textual material and tests for all 3 courses are essentially the same but taught through the discipline. The instructor determines the course grade and a fair cut score for the competency, usually at the B+ or C level. Then all students must pass the CTLE to graduate. Math, Science, Critical Thinking, and Oral Communications will largely follow the Computer Literacy format. Math and Science may use 4-5 courses and will have a graduation exam like the EEWP and CTLE. They will identify and imbed the 20 components into courses for critical thinking and oral communications, but they feel that these competencies are less susceptible to quantitative assessment. They are developing rubrics for these two that may for example require and assess a speech as part of the class final. 4. Do you have in place a system of monitoring student progress and intervening when required? Writing has been a required competency for 20 years prior to the state mandate and it is very developed and more or less stands alone or apart from the other 5 at this point. Instructors monitor student progress, and there is a Writing Center on campus, where students are encouraged or mandated to go for help. A number of writing workshops are also offered throughout the school year to assist students. The competency courses instructors are expected to monitor student progress in all of the competencies, and there is an Assessment Center on campus where students can get information. Intervention and assistance, other than writing, are currently with the faculty and students during the courses. 5. What problems have surfaced that we can avoid? Getting the faculty behind this. The Assessment Center works with each department to establish the 20 components to serve as the template. Then there is course identity and selection and instructor appointment. How to reasonably and validly assess the competency as far as uniform exams in the courses and as far as the EEWP and CTLE type exit exams is an ongoing problem. In writing, for example, there is the matter of having 2 or 3 evaluators from faculty and administration from across campus for the EEWP. They did not have an answer for how to avoid these problems; just have to seek broad input and cooperation. Academic freedom comes into it. They have been fortunate to have had good input and cooperation, but it takes time and patience, and they are very favorable to have an assessment and writing center with director and staff coordinating everything. Mr. Pickering is full-time and has a full-time analyst, a graduate assistant, and part-time of 3 faculty members. The student problem is mainly about the straight-A student who cannot pass the EEWP. Currently just has to go to the Writing Center and retake the exam to get degree. Entering freshmen and transfer students can test out of certain competency course, but if they do not test out, then all must successfully complete the courses to graduate. 6. How autonomous are the individual faculty within the structure? Is faculty responsible for the course design of impacted courses? There is some autonomy but within parameters. Once the competency components have been devised, standard text and testing material are put into place for all courses on a given competency. The autonomy would be in how the material is presented and evaluation of the student's progress and results in the course. The individual instructor is responsible for the design, but the course content must include all of the components required for the particular competency together with the required textual and testing materials that have been selected for the competency. 7. Is the competency approach actively promoted as part of the university identity? Writing still is, because it was emphasized for Old Dominion for 20 years before the state mandate. In state, the other competencies are not really actively promoted, because all are required of all state two- and four-year institutions. All are actively promoted as far as out-of-state prospects, since the approach is still considered unique, especially on a state-wide basis. 8. What effect has the program had on enrollments? The writing competency has always been considered to have a positive impact, although measuring the extent of the effect is difficult. Have always had good feedback from stakeholders. It is too soon to really know what the impact will be on having all of the competencies. The emphasis has been on getting them in place. Writing and computer literacy are up and running. Math and science are in progress for 2004-2005, and critical thinking and oral communications are set for implementation and completion in 2006 and 2007. | 