Basic Research Assessment Results 2006-2007

Geneseo Learning Outcomes in Basic Research

Students will:

• Locate information from a variety of sources.
• Evaluate information from a variety of sources.
• Synthesize information from a variety of sources.
• Support a well-organized argument with this information

SUNY Learning Outcome in Basic Communication (Research)

Students will

• research a topic, develop an argument, and organize supporting details

Method of Assessment

Geneseo assesses four separate outcomes under the heading of "Basic Research." The first three of these provide assessment results for two of the outcomes in SUNY's category, "Information Management." The fourth - Support a well-organized argument based on researched information - provides assessment results for SUNY's one learning outcome under "Basic Communication (Research)."

In spring 2007, Geneseo faculty in 16 major programs identified courses in which their students conduct research. Samples of student work demonstrating student skills with respect to the outcomes above were collected and assessed using the Basic Research rubric. Faculty were asked for the observations on the results collected within their own departments. A selection of these is reprinted below.

Results

Observations from program assessment coordinators

In the comments below, program assessment coordinators reflect on the results collected in their own departments.

Virtually all our students can locate and evaluate appropriate sources for research projects. They have more difficulty developing a question/thesis statement for a research project and synthesizing the information from a variety of sources into effective support for their analyses. About 2/3 of the students exceed or met expectations in relating sources to each other or the main thesis of their paper and the other third are approaching that goal. Only one student was rated as not meeting the "synthesize" standard. While more than half the students did met or fulfill the "argue" standard, nearly 25% of them did not met this goal. This is clearly an area of instruction that needs attention.

Increasing the consistency of the writing assignment across all sections of biology seminar (or other biology courses if necessary) should help to improve the student's results on the assessment. Not all instructors place the same emphasis on having the students develop a clear, focused, thesis topic. They don't make this and the ability to use the primary literature to provide evidence or support for the thesis statement primary goals in the writing assignment. Their emphasis is often more on helping students understand the highly technical details found in the scientific literature. Providing the students a better foundation on highly technical topics is an important goal that may only be achieved by adding and evaluating this type of writing assignment from a 300 level biology elective rather than a 1 credit seminar. In some settings students need to be held more accountable for developing a focused thesis and making well-supported arguments. Rigorous grading including revisions are also critical for the ultimate success in the assignment.

Although the students have taken INTD 105 where they are expected to demonstrate writing skills including developing a thesis and correctly referencing sources, not all of them appear able to transfer these skills to the Biology seminar. Perhaps INTD 105 should draw students' attention to the need to retain and transfer skills learned there. For our part we need to hold students accountable for what they should know and make sure the can recognize new situations where this knowledge applies.

Students are skilled at completing independent research (including locating information from a variety of sources such as texts, journal articles, internet sources, first person testimony, informational flyers, etc). Their sources of information are not exclusively from the realm of communicative disorders which has made this learning experience so rich. The organization of the main points and supporting details within the paper need continued refinement.

The following can be reviewed further in class: main point/supporting detail organization, the roles of the introduction and conclusion, and use of connectives to join thoughts and paragraphs.

Overall, the students assessed were able to find, evaluate and communicate their scientific findings effectively. However, their abilities to locate, evaluate and synthesize the information appear to more effective than their ability to make clear, organize and concise argument based on the information they gathered.

Since their overall abilities to locate and report scientific data were acceptable, no major changes are planned. However, we might consider paying closer attention to the way in which our students argue or express themselves in the laboratory or research reports that they submit each semester.

In the introductory class, most students applied basic APA format correctly in their final research papers, after having had an opportunity to write a topic paper and receive feedback earlier in the semester. The basic ability to use APA was also demonstrated by most students in the advanced class.

In the introductory class, most students cited reputable sources and met or exceeded the number of sources for the research paper; however, fewer were able to evaluate and synthesize the information and apply it to the main thesis of their projects. This basic trend reappaered in the advanced class, where students can generally find sources but are not as strong using that information to put together a strong argument for a particular position. Many advanced students were uncomfortable not having a specific number of sources to find and struggled with the idea of simply finding the necessary sources to support a sustained argument.

In the introductory class, a larger than expected number of students either did not meet, or only approached an adequate level of clear writing and general organization of a research paper, even with the prior experience of the topic paper. Fundamental errors of spelling, grammar, and syntax were noticeable on many papers. It is likely students did not review and edit papers before submitting them. These issues were not as evident in the advanced class, possibly because the final research proposal was a cumulative paper that students developed over the course of the semester.

In more of the classes in the Communication department, it may be important to create more assignments or in-class exercises that reinforce and continue to develop writing skills, as well as the ability to clearly state, organize, and evaluate theoretical concepts. Every one of these need not be formally evaluated, but some form of feedback and assessment is needed to encourage development.

The rubrics that are used to assess the topic and research papers may need to weight writing (and possibly APA) standards more heavily. Until these have a significant value placed on them during the grading process, many students may not take them seriously.

Writing clearly and developing a well-organized argument are basic skills students should develop in high school, and certainly in college. Most students in our introductory course (and certainly our advanced course) have completed INTD 105 (and some INTD 101). Perhaps more attention should be given in these courses to reinforce skills of correct, clear writing.

Over 1/4 of the students met or exceeded standards for locating and evaluating information. Not surprisingly, fewer students met or exceeded standards for synthesizing the information and using it to support a well-organized argument, in that the course for this assessment is at the beginning of the program. A consistent 5% of students did not meet standards.

Professor Reynolds (Music): The students generally did well because the step-by-step process I took them through, in which they were evaluated at each stage, pretty nearly guaranteed success by the end. . . . the move to the next stage depended on success in the first one, and no one moved on until they'd completed that stage successfully. I consider this to be one of the successes of the semester: that all of my students completed the various stages of research at a level of depth appropriate to college.

This assessment reviewed research papers submitted for Phil 398 (Seminar: major Philosophers, Liebniz). This is a required course for all Philosophy Majors. As a culmination course for these majors, it is a good measurement of how majoring in Philosophy prepares students in research-skills areas. The main finding is that most Philosophy majors (12-13 out of 14) meet or exceed in Basic Research skills.

Professors Board and Bosch assessed the students in different areas that form part of completing a successful research paper i.e. defining topics for research, using specialized indexes for finding bibliography on the topics, developing critical approaches to the topics, assesing previous scholarship on the topics, developing an argument for the research paper, organizing the research paper, listing bibliography, writing in good standard English. While the majority of students were successful in most of these areas, only nine out of thirty were successful in all of the areas and ten consistently underperformed in all of the areas. It is clear that students are not coming to college properly prepared to successfully take on complex research and writing. They are also having problems in recognizing arguments in the articles and books that they read and in constructing their own arguments.

Students who have not been properly prepared by a school system that consistently leaves them free to avoid writing and thinking in a cogent and original manner need remedial help to address the main problem that a faulty educational system has given them. Students who have poor reading and writing skills and who cannot conceptualize because they have been oriented towards a concrete goal-obsessed multiple choice approach cannot shift their skills when confronted with advanced demands. Since Geneseo does not require attendance and the system is ever-ready to give students an "out" a message is given that it is okay to not show up for class and that it is okay to not turn in assignments on time. Perhaps assessment should be more student-directed and its goal should be to make standards clear to students and they should be consistently given information about how they are not meeting these standards. Remedial witing and reading classes taught by faculty trained to teach these areas would help. A list of books that could provide students with much-needed historical and cultural contexts and background would also help. Expectations should be raised across the board at Geneseo and students who cannot meet these expectations should experience the consequences of their inactivity, lack of interest and lack of performance. We should be able to penalize students who don't show up and students who lack basic skills should be told this point blank and students who lack basic knowledge should be tested and shown where they fall short and given the chance to develop themselves as they are assessed and measured against their peers and their own progress.

It is time that Geneseo gave a consistent message that students need to stretch themselves and that they need to take responsibility for their knowledge, their performance and their education. Faculty can point the way but if students are not being required to even come to class, the message being given is that faculty and what they do is not worth the respect and attention that steady attendance requires.

Generally our students appear to be doing very well in basic research. A significant number of the papers we read were top-notch research projects that utilized extensive primary and secondary sources (including a number of papers that integrated archival research), and that articulated creative and persuasive arguments. On the other hand, we found very few students whose work fell into the "not meeting standards" category.

Under "Locate information from a variety of sources", we found room for improvement. Our students seem to be able to locate a variety of secondary source materials (journal articles, scholarly monographs, etc.) and it appears that they are relying heavily on Information Delivery Services (IDS) to bring these secondary sources on to campus. Students seem to be less successful in integrating primary sources into their papers, and the committee felt that the limited range of primary sources available to students on campus was a significant liability. Virtually all of the papers that either "met the standards" or "exceeded the standards" were based on research off campus - primarily through materials available at the University of Rochester's Rush-Rhees Library (for example, microfilm collections, rare books, etc.). A number of these papers reflected particularly ambitious research, including work with archival materials and oral histories. For work that was "approaching the standards" , we found that almost all of the students were using some primary sources, but often were relying on a limited number of locally-available primary sources or too many secondary sources. It should also be noted that the results in this category may be affected by the nature of specific research topics - students working on projects covering certain areas (for example non-English European or non-western topics) or time periods (for example Classical or Medieval) may not have access to a deep body of sources even when they use IDS and the UR library.

In assessing "Evaluate information", we were very pleased to see that our students are doing a very good job of differentiating between reliable and unreliable sources. We found very few examples of students who used obviously problematic sources (for example, the uncritical use of websites), and all of the students who either "met the standards" or "exceeded the standards" had some direct critical discussion of the primary and secondary in the body of the paper or annotated bibliographies. In many of the "approaching standards" papers, we felt that the addition of just a paragraph of critical reflection on the strengths and weaknesses of sources would have resulted in significant improvement. We believe that this is a direct reflection of the substantial amount of attention that is paid to primary source criticism in the HIST 221 course in particular.

Under "Synthesize information", we found that our students are also doing a reasonably good job of explaining the significance of their research, how their primary sources relate to the scholarship of other historians, and how this research relates to the thesis of their work. In assessing "Support a well-organized argument with this research", the committee found that most of the papers were well-structured and contained an historical argument based on reliable research. Many of the papers in the "approaching the standard" category contained a critical thesis that was buried in the body of the paper or that was presented in a manner that partially obscured the point. Although in most cases an argument did emerge as the paper progressed (we found very few examples of papers that had no thesis whatsoever), we felt that these students could be doing a better job of clearly and specifically articulating a thesis throughout their papers.

In reviewing the results for the "support a well-organized argument" category, the committee found that low marks in thos categories often correlated to lower performance in the "Locate information..." category. That is, papers that were based on a small body of primary sources also tended to have weak argumentation and a poorly-supported thesis. In papers that were "approaching the standard", we encountered a number of projects that clearly would have benefited from additional research or that would have likely progressed in very different directions if the students had accessed a more diverse body of primary source material.

This assessment reveals that although our students are doing well in identifying and integrating secondary sources into their research projects, some are not as successful in identifying a sufficiently deep and diverse body of primary sources. The bibliographies to the student papers suggests that student research in history is heavily reliant on IDS and resources that are only available off campus (particularly the University of Rochester), and that students who do not utilize these off campus resources may have difficulties in creating a primary source base of sufficient depth and diversity. These papers seem to manifest a variety of problems, including weak argumentation and poorly developed thesis. Although the paucity of primary sources on the Geneseo campus is likely an insurmountable issue, the committee felt that faculty could explore several strategies for reminding students of the importance of primary source research in history papers and the resources that are available locally (particularly at the University of Rochester and other area colleges and universities).

First, it might be beneficial for faculty in courses with research projects (i.e. HIST 220 and 221, upper level electives, and the senior capstone experience) to structure a specific assignment that asks students at mid-semester (or earlier) to identify relevant and accessible primary sources. This would encourage students to think about primary sources early in the project and would emphasize the importance of original research in a history project. Second, faculty could remind students of the rich holdings available at Rochester in all classes where research projects are required and could be proactive in providing students information on campus services that make a visit to UR more convenient (particularly the weekend bus service to Rush-Rhees Library). Where relevant, it might also be helpful for faculty to provide students with suggestions of specific categories of materials (for example, a handout on specific microfilm collections that students might consult) that are available at UR or create assignments that require students to at least explore the UR online library catalog. Third, the department might look into coordinating structured field trips to Rush-Rhees, perhaps with the support of student led groups such as the History Club. We suspect that some of this is already being done in classes (particularly HIST 220, 221 and 391), but additional attention and reinforcement in 300-level electives might raise our results in the "Locate information..." and related categories.

It is also clear that some of our students are struggling when it comes to articulating the relationship between their research and their argument. Most of the papers we reviewed seem to reflect extensive work on revisions under the guidance of faculty. Within this process, it might be helpful if faculty directed more specific attention to the crafting of a clear thesis statement, to the introduction and conclusions of the papers, and to transitions in the body of the paper (particularly sections that explain the significance of evidence). This is again something that we suspect is already going on in most classes, but we could possibly do more to emphasize the importance of these key components of written work when we provide feedback on drafts.

Broad exposure to sources and methodology through assignments did not necessarily yield excellent final research papers. Field-based research projects were varied, focused, and well-supported by targeted evidence; however, students follow-up work in some instances fell short of expectations.

Final papers/projects should be more explicitly linked to prior assignments, lecture content, and readings. Regarding field-experience research: more lead time and preliminary meetings, tighter pre-field reading requirements, tighter post-field supervision of finished work.

Students seem more adept at consuming knowledge rather than producing (and discussing/analyzing) meaningful results from"raw" evidence. We need to build a stronger bridge that links primary evidence to relevant questions, ideas, and approaches. Evidencedoes not necessarily "speak" to students with the same clarity that it does to experienced scholars.

How do we encourage students to avoid (or shed) a philosophy based on a too-low commitment of thought, interest, and effort? What happens to Freshman enthusiasm and receptivity? Are upper division "core" courses particularly subject to the ethic of "just enough" work targeted at a passing grade?

Mathematics students are, generally, prepared to gather information for a research project and report their findings coherently. Additional instruction appears to be needed in preparing a document using the APA stylesheet and in properly citing sources.

Students do a good job of locating relevant sources, and finding the connections between them. However, students are not very skillful at using these sources to develop a fresh argument. Instead, the arguments are mainly themes that are taken directly from the sources.

Instructors in PHYS 341 could emphasize that students are permitted to critique the sources, rather than accepting everything they find in them.

Students need to learn how to distinguish between reputable, genuinely scholarly sources and inappropriate ones such as Wikipedia. Students need to improve their skills in writing well-organized arguments.

It might be helpful for instructors to provide two grades for each written exercise, one having to do with the student's basic research and the other having to do with the student's written work.

It would not be surprisng if students in other Gen Ed areas need to learn how to distinguish between reliable and unreliable sources and if they also need to learn to improve their skills in writing well-organized arguments. Instructors in other areas might also provide two grades for each written exercise.