• No labels

1 Comment

  1. Department of Communication

    Closing the Loop on Effective Research Skills


    Closing the Loop – What does that mean?

    “Closing the loop refers to the action of making informed program changes, as needed, based on whatever assessment process has revealed about student learning in the program.” (Source: Academic Assessment- Geneseo Wiki)

    “In recording yearly assessment results, department chairs and assessment coordinators are encouraged to provide extended and concrete discussion of loop-closing action that their programs have taken or plan to take.”  (Source: Academic Assessment- Geneseo Wiki)

    2009-2010 Assessment Year – Effective Research Skills

    The Department of Communication assessment committee assessed the learning outcome of effective research skills.

    • This is one of six outcomes the department has decided to assess periodically, either one or two outcomes per academic year.  The other outcomes are: effective speaking skills, effective writing skills, co-curricular participation, theoretical understanding, and theoretical application.
    • The outcome of effective research skills is aligned with the SUNY General Education Assessment outcome of basic research skills; in the same way, effective speaking skills is aligned with the SUNY General Education Assessment outcome of oral presentation and evaluation.
    • The assessment committee measured effective research skills (basic research skills) in three ways:  First, writing assignments from an introductory Communication course (Comn 160- Introduction to Mass Communication) were evaluated.  Second, writing assignments from one 300-level advanced course (Comn 362 – International Mass Communication) were evaluated.  Third, final papers in a research-intensive class (Comn 368 – Research in Media and Cultural Studies) were evaluated.  Data were collected in fall (Comn 362) and spring (Comn 160, Comn 368) semesters of academic year 2009-2010.  The approach to data collection permitted assessment of the differences between students in introductory and advanced classes, and the evaluation of results in a required introductory course of the program (Comn 160).
    • Assessment results led to the following observations: First, two-thirds of the students assessed in Comn 160 and Comn 362 either met or exceeded effective research skills (basic research skills) across four categories of evaluation (“Locate,” “Evaluate,” “Synthesize,” and “Argue”). Second, when comparing results across the four categories for introductory and advanced courses (Comn 160 and Comn 362) there was observable improvement of research skills for most students even as writing and class assignments were more demanding in the 300-level course.  Third, scores of research skills, across all four categories, for Comn 368 were noticeably higher than in the other two courses that were evaluated, likely because of the GPA and other prerequisites required to enroll in the course, and its specialized research focus.  Fourth, there were greater standard deviations across all four categories of measurement (“Locate,” Evaluate,” Synthesize,” and “Argue”), in Comn 160 as compared to Comn 362, likely because in the required 100-level course the student population ranged from the freshman to senior levels.  Detailed information is found in the department’s 2009-2010 assessment report on basic research skills. 

    An Historical Overview - Effective Research Skills, 2003-2009

    • In academic year 2003-2004, the department added an outcome dealing with research as part of its ongoing assessment plan.  The outcome, as it was stated then read: “Students will demonstrate an understanding of the role of research in the study of communication.”  The department’s aim was to develop a pilot project to evaluate this outcome in response to SUNY Geneseo’s General Education assessment of basic research.  Faculty shared their preliminary observations of students’ understanding of the role of research in the discipline based upon the content of student papers in 300-level courses like Comn 364 (Seminar: Issues and Problems in Broadcasting). Instructors assigned to teach Comn 135 (Introduction to Communication Theory), also monitored how students applied research and library skills when writing papers in that course.
    • By year 2005-2006, the outcome, now one of six identified by the department, was redefined as the ability of students to “…demonstrate competency in the process of locating and assessing material for use in a written or oral report.”  The intent of the department was to make this outcome a part of ongoing assessment in support of the periodic General Education assessment of basic research, to simultaneously align the outcome with student competencies in oral presentation and written communication, to determine whether a foundation of research skills, principally locating appropriate sources, was achieved in Comn 135, and to determine if competent research skills were observed by instructors in 300-level courses.   
    • Effective research skills (basic research skills) were assessed in academic year 2006-2007 as a part of the institution’s General Education assessment of this outcome. It was found that at the introductory course level most students were able to cite reputable sources and applied APA format correctly, but fewer were able to evaluate or synthesize research data when applying it to their writing projects.  A similar trend also appeared at the advanced course level.  A larger than expected number of students either did not meet, or only approached an adequate level of clear writing and organization in the introductory course.  This deficiency was not as pronounced in advanced classes.  Still, it was generally found that students could effectively locate appropriate sources, but were less skilled at using research data to write or orally present strong arguments.  These results underscored the close relationship between research and oral and written communication competencies.  Recommendations of the department to improve effective research skills, including writing competencies, in year 2006-2007 were: 1) increasing the number of in and out of class assignments that reinforce organized presentation of ideas, 2) weighting clear writing and proper use of APA on research assignment rubrics more heavily, and 3) offering more instructor feedback, formal and informal, to encourage development of students’ research skills.
    • Because the department recognized the integral relationship between research and communication competencies, the learning outcome of basic writing skills was assessed in 2007-2008, expressed as “competency in the effective presentation of original ideas and supporting material.” Results in introductory and advanced courses showed that the majority of assessed students (approximately two-thirds) met or exceeded expectations of choosing and critiquing appropriate sources for written assignments (“Evaluation”).  Students were less skilled, however, synthesizing and writing cohesive arguments.  Furthermore, a high percentage of students in advanced courses acquired skills of evaluation and synthesis (80%-96%), but there was minimal difference between 100 and 300-level students regarding the formulation and support of cohesive arguments. The assessment category of “Argue,” relevant both in the evaluation of effective research skills and effective writing, remained a point of concern.  Undergraduate majors, it was found, “…may be able to write good descriptive essays based on their library research, but some [were] uncomfortable producing their own argumentative essays.”  From these data five suggestions surfaced:  1) expectations of student writing competencies should be assessed according to overall student GPA level, or perhaps credit-hour level, 2) student writing should continue to be emphasized and assessed across all course levels, 3) instructors may need to clearly and repeatedly voice their expectations of writing skills, 4) given the mix of freshman through seniors in the department’s introductory classes, those eligible to enroll in 100-level courses should be examined, and 5) renewed attention should be given to the category of “Argue,” in future ongoing assessment, whenever evaluating “effective research skills,” and “effective writing skills.”
    • Academic year 2008-2009 was devoted to department program review.  With that in mind, department faculty reviewed and discussed assessment procedures, the list of outcomes, and results achieved as an important aspect of self-study.

    Closing the Loop on Effective Research Skills

    Ongoing assessment of the learning outcome, effective research skills, has produced a number of results:

    •  The addition of effective research skills as a learning outcome of the department, followed by refinement of its definition.
    •  Alignment of the four evaluative categories for measuring effective research skills with institutional assessment of  basic      research skills.
    •  Recognition that, among our majors, research skills of synthesis and argument lag behind those of locating and evaluating appropriate sources.
    •  Encouragement of faculty to place more emphasis on assignments that require different types of research appropriate to course level,
    •  An instructional need to unmistakably and repeatedly communicate expectations for effective research and writing in class assignments.
    •  Changes to research-intensive courses at the 300-level to improve research skills of junior and senior majors (e.g., Comn 368, Research in Media and Cultural Studies).
    •  The addition of new research components in Comn 277 (Online Journalism) and Comn 253 (Media Advertising).
    •  Recognition that assessment of  “effective research skills” has been measured more in one instructional concentration of the department than others, namely the Journalism and Media track. 
    •  Recognition that the measurement of research skills involves assessment of competencies of written and oral communication. 
    •  Emphasis on locating reputable sources and use of APA format in the introductory courses required of all majors (Comn 102, Comn 103, and Comn 160).  Similar emphasis on the organizational principles students must apply when writing or presenting arguments.

    **Lastly, the “closing the loop” report is being used as a means to seek feedback from faculty members and stimulate further discussion of questions or issues relevant to the assessment of effective research skills.