NASA Science Mission Directorate:
Earth Science Education Product Review

Overview

Since 1995, NASA's Earth science education program has sponsored independent peer reviews of its education learning resources. The purpose of this review is to ensure that Earth science education products distributed by NASA are of high quality and meet rigorous standards. The review is based on the learning resources’ relevance to NASA’s education mission and Earth system science program, scientific accuracy, pedagogy (educational value), effectiveness of presentation, documentation, ease of use, and power to engage and/or inspire the target audience.

To ensure an impartial, objective review, an outside organization, the Institute for Global Environmental Strategies (IGES), organizes and manages the review process.

Products submitted to the review that were developed or supported (funded) by NASA are accepted into the review process. Additional products developed outside of NASA are considered on a case-by-case basis. Before being accepted to the review, non-NASA education products need to demonstrate that they support NASA’s overall mission and that the product has a strong relevance to NASA’s unique contribution to learning about Earth system science.

Submissions to the review range from printed to electronic formats, and cover both formal (elementary-college level) and informal education products. It is expected that products reviewed will be in final development prior to submission, as this review is not intended to replace field testing and formative evaluation.

At the beginning of each review cycle, a call for products is released to NASA representatives involved in Earth science education. Distribution includes all NASA Field Centers along with universities and other organizations receiving grants or on contracts with NASA for the development/implementation of education materials. The call is also announced in the Earth Science Education update monthly newsletter, and distributed directly through its listserve. Products are due in late November/early December for the Fall/Winter review, early May for the Spring/Summer Review.

After the products are received by IGES, review panels are identified and selected according to the type of product, education audience, level, and content area. A typical panel has 5-7 reviewers, including: classroom teachers with experience at the appropriate level; education specialists who are familiar with the current trends and needs in science education and knowledge of national education standards (e.g., science, math, geography, and education); informal educators; and scientists with a relevant background in the science content of the materials being reviewed.

Review Process

Reviewers receive packages, which contained instructions, evaluation forms, and the products to be assessed. Also included are the product submission forms for each product, which provide additional information about the products including a description of the product, a description of how the product has been reviewed for educational value and scientific accuracy, audience/level/format, subjects that the material covers, key messages, goals, and objectives, and contact information for the product developer and point-of-contact for the review. Developers are encouraged to provide additional background information in their submission form that they believe is important for reviewers to have.

To ensure a consistent approach to evaluating the products, instructional teleconferences are conducted after reviewers receive their packages and are a requirement for all new reviewers. During the teleconferences reviewers are provided background information on the review process, general instructions on how to proceed, and an overall context for the review. The teleconference also provided an opportunity for reviewers to ask questions regarding the evaluation instruments, the products, or expectations.

Following the instructional teleconferences, reviewers ha\ve approximately six weeks to conduct their individual assessments using the evaluation forms provided. IGES is available for consultation throughout this stage. For example, if a reviewer has technical difficulties or questions about the completeness of a product, IGES responds to those inquiries, acting as a liaison between the reviewers and the product developers. Reviewers do not have direct contact with the product developers.

Reviewers submit their completed evaluation forms to IGES, who then compile reports for each product, which serve as the basis for the panel discusssion. These reports include a summary, as well as the complete individual reviewer forms for each product in a panel. The summary reports identify all of the individual reviewers' ratings and recommendations for each product, as well areas of agreement and disagreement. Each panelist receives a copy of this report prior to their final, panel teleconference, which allowed them to review their own, and others, comments and completed forms.

The final step for reviewers is to participate in a panel teleconference. The teleconferences, convened by IGES, allow panel members to discuss the products and their evaluations, and formulate a consistent group recommendation to NASA. By providing the reviewers this opportunity, those with differing opinions are able to determine a recommendation that is acceptable to all. Reviewers also provide feedback on the review process, including the evaluation forms, procedures, and products.

Following the panel teleconferences, IGES prepares a consolidated report of the panel recommendations, which is presented to NASA Earth science education program managers, who reviewed the panel recommendations and determine the final recommendation for each product.

The results of the review are used in helping NASA determine which products are ready for distribution, and the recommended distribution method. Recommendations may range from broad distribution, such as at national conferences, to distribution through teacher workshops, to limited distribution, such as through NASA Central Operation of Resources for Educators (CORE), teacher workshops, or to limited audiences (for example, a specific geographic region or audience). Reviewers may also recommend that a product needs revisions before it is distributed by NASA or not recommend that NASA distribute the product.

Participating in the review has several significant benefits, including:

• NASA Earth Science Education Catalog Listing. Products that pass the review are listed in the NASA Earth Science Education Catalog.
• DLESE. Products that pass the review are also added to the Digital Library for Earth System Education (DLESE) at http://www.dlese.org as part of the NASA-reviewed collection – a select group of education resources that have passed the NASA review.
• Distribution by NASA. Products that pass the review will be considered for distribution by NASA at national education workshops and conferences such as the National Science Teachers Association (NSTA) and the National Council of Teachers of Mathematics (NCTM).
• Feedback from education and Earth system science experts. While this review does not take the place of field testing and formative evaluation of education products, it will provide independent feedback to product developers.

For more information on the review, please contact: Theresa Schwerin, Institute for Global Environmental Strategies (IGES), theresa_schwerin@strategies.org.