Supplemental Information:

n/a

Assessment Type:

observation tool

Scale:

4 level rubric with qualitative descriptors for each level

Publication Date:

Jun 04, 2012

Respondent:

4th-8th graders

Domain(s) Evaluated:

Engagement, Attitude / Behavior, Content / Knowledge, Competence, Career Knowledge / Acquisition

Sample items:

EVIDENCE ABSENT	INCONSISTENT EVIDENCE	REASONABLE EVIDENCE	COMPELLING EVIDENCE
There is minimal evidence that students are engaged with hands on or interesting activities where they can explore STEM Content.	There is weak evidence that students are engaged in hands-on or interesting activities where they can explore STEM content.	There is clear evidence that the students are engaged with hands on or interesting activities where they acan explore STEM content.	There is consistent and meaningful evidence that students are engaged with hands-on or interesting activities where they can explore STEM content.
1	2	3	4
The activities mostly leave students in a passive role, where they are observing a demonstration or listening to the facilitator talk (minimal hands-on opportunities).	Students engage in hands-on activities; however, there is limited evidence that the hands-on activities encourage students to engage with STEM content in meaningful ways ("hands-on, minds-off").	There are some opportunities for students to engage in hands-on activities that allow them to actively explore STEM content. Some parts of the activities still leave students as passive observers while the facilitator does all the cognitive work.	There are consistent opportunities for students to actively explore STEM content by engaging in hands-on activities, where students do the cognitive work themselves and the facilitator maintains the role of a facilitator versus teller.

Reliability:

Validity:

inter-rater reliability established

Frequency:

Rarely

Administration time:

60 minutes

Requires a Computer:

No

Requires Internet Access:

No

Primary reference:

Still under development by the Program in Education, Afterschool and Resiliency (PEAR).

Comments:

To use the tool, training is required (available in person or via webinar), and then program administrators or staff can use it internally to monitor the quality of their STEM programming, or can hire evaluators to use it to provide feedback and scores.

Other Reference:
Shah, A. M., Wylie, C. E., Gitomer, D. & Noam, G. (2014). Development of the Dimensions
 of Success (DoS) Observation Tool for the Out of School Time STEM
 Field: Refinement, Field-testing and Establishment of Psychometric Properties. Retrieved from: http://www.pearweb.org/research/pdfs/DoSTechReport_092314_final.pdf
The instrument had inter-observer reliability to gather consistent data, with agreement from Kappa values. Moderate levels of agreement for four dimensions, with fair for six and slight for the last two. Exploratory factor analysis showed dimensions divided into two groups (context and meaning making).

STEM Criteria

Science

Yes

Technology

Yes

Engineering

Yes

Math

Yes