National Center for Professional and Research Ethics
Year 1 Evaluation Plan (January 15 – September 15, 2011)
January 30, 2011
Jennifer C. Greene & Gabriela Juarez, Internal Evaluation Team
The National Science Foundation (NSF) has funded the development of an online resource center for the responsible and ethical conduct of research (RCR) to “help educators, researchers, undergraduate and graduate students, postdoctoral fellows, and practicing scientists and engineers understand the scope of existing knowledge about both ethics and ethics education in science, mathematics, and engineering, as well as to develop analytic approaches to the study of ethical issues in science, mathematics, and engineering in both national and international contexts” (RFP, p. 3). NSF has long supported ethical conduct in research and ethical training for students and postdoctoral fellows, expecting all grantees to be knowledgeable about the ethical conduct of research. Section 709 of the America COMPETES Act makes this training requirement explicit in calling for “easy access to high quality case studies, best practices, and original scholarship in ethics in science, mathematics, and engineering” (ibid). The establishment (creation and maintenance) of this online ethics resource center is NSF’s response to this legislative requirement. It is intended to be broadly applicable to all fields that NSF supports – engineering, mathematics, natural sciences, social sciences and computer sciences. Although there is a variety of extant resources that address ethical issues and concerns (including books, videos, websites, online courses, and journals), there remains a need for a dynamic, sustainable online resource that incorporates all of these existing resources, as well as new material developed or procured by the project team.
The center is intended to be populated with creative, interactive materials that facilitate users’ learning of the pedagogy of ethical training and the enactment of ethics in practical research contexts, both in the laboratory and in the field. The center is further intended to be dynamic, enabling materials and research findings to be updated and added as they become available. And center is expected to be sustained beyond the 5-year time period of this project.
NSF has funded the development of this center as a cooperative agreement. The project team sets measurable outcomes (deliverables) for each year, which are then approved by NSF. Renewal of the project for the subsequent year depends on accomplishment of the activities defined in the yearly outcomes.
The Ethics Collaborative Online Resource Environment (Ethics CORE) at the University of Illinois at Urbana-Champaign
The proposed Ethics CORE project housed at the University of Illinois at Urbana-Champaign fulfills all of the parameters set forth in the NSF request for proposals. It is designed to provide information and expertise on the ethical conduct of research for instructors, students, field researchers, administrators, scholars of ethics in research, journal editors, and other audiences who may find the material relevant and useful.
The plan for the Ethics CORE is anchored in a contemporary technology (the HubZero platform), a thoughtful educational plan for both pedagogy and content, and significant additional content partners (notably including NAE, PRIM&R, and AAAS).
Key Evaluation Questions for Year 1
The ultimate success of this project is defensibly judged on the basis of use, user feedback, and the quality of ethical teaching and conduct in the field. In this initial developmental stage of this project, the evaluation team can likely be most critically helpful by providing a framework within which the PI team and broader Steering Committee can monitor both the accomplishment and the quality of year 1 tasks. Quality considerations will focus on how well year 1 task accomplishments address identified dimensions of importance, value, and quality in the overall Ethics CORE project vision.
More specifically, for this first developmental year, the evaluation will focus on three primary considerations.
1. How well does the design for the project address current needs for resources on responsible and ethical conduct in research for the full range of diverse potential users?
This question signals the participation of the evaluation team in the needs assessment activities to be undertaken in this first year of the project.
2. How well are year 1 tasks accomplished, in terms of addressing the identified criteria of quality for the overall project?
The second evaluation question will require the identification of “key criteria of quality” for the various components of the project.
3. To what extent and in what ways do year 1 activities and accomplishments advance the inclusion and participation of individuals and groups traditionally under-represented and underserved in STEM fields?
And the third question will be addressed via interviews of project personnel and key informants in the field and via reviews of project materials.
Design, Methods, and Timelines
Brief initial plans for data collection and analysis for each evaluation question are presented in this section. Timelines are included, as relevant.
Q1. How well does the design for the project address current needs for resources on responsible and ethical conduct in research for the full range of diverse potential users?
The evaluation team’s participation in the project’s needs assessment will take three forms. The evaluators will (a) review the project’s needs assessment plans for inclusion of a range of diverse potential users of the Ethics CORE; (b) participate directly in selected needs assessment activities (for example, interviews with on-campus scientists and students); and (c) participate in the review and interpretation of the needs assessment data, in terms of their implications for project design.
These activities will take place in the spring semester of 2011, as part of the project’s planned needs assessment.
Q2. How well are year 1 tasks accomplished, in terms of addressing the identified criteria of quality for the overall project?
The process of evaluation involves judgments of quality of specified project activities and materials. These judgments rest on criteria of quality, commonly identified via combinations of standards in the field, expert opinion, research evidence, and policy or practical expectations. The highly complex Ethics CORE project involves several distinct components, each of which may be usefully judged on its own criteria. And there may be some criteria that are common to or shared by all project components.
The idea behind this developmental evaluation question for year 1 is to encourage core project team members to identify these key criteria of quality and then begin to use them in monitoring and judging project progress, beginning with task accomplishment during this first year of the project.
The evaluation team will facilitate this process of criteria identification, relying heavily on the project team members’ expertise, and will develop procedures by which the criteria can be integrated into ongoing project development, monitoring, and decision making. The evaluators will also work with the project team to assess the quality of year 1 accomplishments according to these criteria.
Work on this task will take place over the remainder of the project’s year 1.
A very preliminary listing of possible criteria of quality, culled from the RFP and from the project proposal, is presented in Appendix A. A next step in developing and refining this set of criteria is to gather the expert views of the core project staff, and perhaps Steering Committee or other experts as well. It may also be useful to translate this list of quality criteria into measurable indicators of quality project progress.
Q3. To what extent and in what ways do year 1 activities and accomplishments advance the inclusion and participation of individuals and groups traditionally under-represented and underserved in STEM fields?
Issues of equity in access, quality of experience, and accomplishment remain central to the American society, and indeed to societies around the globe. Particularly with publicly-funded projects in the U.S., it is critical to ask, who participates and who doesn’t? who benefits and who doesn’t? And particularly in STEM fields – which have a long history of under-representation of individuals from groups traditionally under-served in our society – it is important to ask these questions.
This evaluation team will ask these questions as relevant to the developmental activities of this first project year. The evaluation team will likely use observations, reviews of project materials, and interviews with core project staff and key informants elsewhere to gather data relevant to this question.
Activities for this purpose will be initiated in the spring 2011 semester and continue through the summer.
The evaluation team is fortunate to live and work in the same place as the core project team. So, some evaluative reporting will be in the form of informal conversations as we attend project meetings and events.
More formally, we will provide a bi-monthly “evaluation memo” that addresses one or more key issues in this developmental stage of the project, with data, questions, and perhaps a few insights we hope will be helpful to the core project team. We will also provide a year 1 evaluation report in August 2011.
A Preliminary Listing of Key Criteria for Judging Quality
The technical infrastructure of online resource center meets the following criteria for diverse learners and users*:
Meets accessibility standards
Creative, innovative, includes multiple media
Offers strong educative potential
Current in the use of available hardware and software
Expandable, extensible, customizable
Robust (rarely crashes, handles material in multiple digital formats)
The content of ethics resource center meets the following criteria for diverse learners and users:
Wide variety of content for diverse audiences (e.g., course syllabi for instructors, best-practices recommendations for administrators, scholarly articles for researchers, stand-alone modules for students)
Current, up-to-date, relevant
Site will integrate research with education, and will be built on basis of what we know and learn about how students learn about RCR
Innovative, creative, involving multiple media
The pedagogy of the ethics resource center invokes the following learning processes address diverse learners and users:
Socially interactive, contributing to existing and newly-created online communities of practice (e.g., via discussion forums, social networking)
Interactive, connecting people in different silos, one to another
Pedagogically sound, involving active learning
Experiential, using role play, cases, simulations, in-box exercises
The overall online ethics resource center is organic and sustainable in the following ways:
Stimulates creative development of materials by others
Is known nationally and internationally as THE site for information on the ethical conduct of research in NSF-supported fields
Serves as a model for other online resource repositories
Provides for digital preservation through the excellence of the Illinois Library
The online ethics resource center is consequentially effective for diverse learners and users:
The center is well used
The center is well regarded
Users are knowledgeable about responsible and ethical conduct in research
Users are ethical research practitioners
Users demonstrate capacity for context-specific moral deliberation and judgment
Users demonstrate normative understandings of ethics in practice
The center contributes to ethical pedagogy and ethical research practice
The processes used to develop the online ethics resource center:
Involve a fast start-up
Make good use of existing resources
Involve meaningful and active partnerships
Demonstrate good stewardship of the field
Include ongoing dissemination to varied professional forums
* Diversity includes traditional demographic markers, academic discipline, geography, type of educational institution or employer …