Student Outcomes from Innovations in Undergraduate Chemistry Laboratory Learning. A Review of Projects Funded by the U.S. National Science Foundation between 2000-2008*

Resultados en el estudiante a partir del aprendizaje en el laboratorio de química de la licenciatura. Una revisión de los proyectos subvencionados por la Fundación Nacional de Ciencias de Estados Unidos

Hannah Sevian,^1-2 Gavin W. Fulmer³

¹ Department of Chemistry, University of Massachusetts Boston, 100 Morrissey Blvd., Boston, MA 02125, USA.

³ National Science Foundation, 4201 Wilson Blvd., Arlington, VA 22230, USA.

Abstract

Much of the research on implementation of novel curriculum, instruction, and assessment in undergraduate science laboratory courses in the U.S. has been funded by public funding agencies. A review and analysis was conducted of awards made between 2000 and 2008 from the National Science Foundation that have focused on laboratory learning in undergraduate chemistry. The study is concerned with characterizing the types of interventions that occurred, and what was studied and learned related to student learning and associated outcomes. For projects that concluded by August, 2011, an overview is presented of the current 'state of the art' of the empirical knowledge base, in the interest of suggesting gaps and opportunities for further study. The findings are also contrasted with a recent summary of the state of chemical education research, largely focused at undergraduate level teaching and learning, that was presented in a review paper commissioned by the National Academies of Science as part of a consensus study on discipline-based education research.

Keywords: postsecondary chemistry; chemistry laboratory learning; student learning outcomes; research funding.

Resumen

Mucha de la investigación acerca de innovación curricular, educación y evaluación de la implementación en la licenciatura de cursos de ciencia con laboratorio ha sido subvencionada por agencias públicas de financiamiento. Se ha realizado la revisión y análisis enfocados a los otorgamientos hechos por la "National Science Foundation" entre 2000 y 2008 dirigidos al aprendizaje de química en la licenciatura. El estudio se dedica a caracterizar los tipos de intervención educativa que se han presentado, lo que fue estudiado y sus resultados relativos al aprendizaje estudiantil y otros logros paralelos. Para los proyectos que se completaron en este periodo de nueve años se presenta una panorámica del 'estado del arte' actual de la base empírica de conocimiento, con el interés de detectar brechas y oportunidades para el estudio ulterior. Se contrastan igualmente los hallazgos con un resumen reciente del estado de la investigación educativa de la química, centrado en la enseñanza y el aprendizaje a nivel licenciatura, que fue presentado en un artículo de revisión comisionado por las "National Academies of Science" como parte de un estudio consensual sobre investigación educativa basada en las disciplinas.

Palabras clave: química universitaria; aprendizaje en el laboratorio químico; resultados del aprendizaje estudiantil; financiamiento de la investigación.

Introduction

Chemistry is a laboratory-based science. Thus, learning in the laboratory is critically important to developing knowledge of chemistry, whether a student ultimately seeks to become a professional chemist, to follow a career path that requires some understanding of chemistry, or simply to be a scientifically literate citizen. Laboratory-based courses or course components accompany nearly every chemistry course at the undergraduate level, both for science majors and non-majors. It is, therefore, no surprise that there have been many efforts by faculty in chemistry departments to develop courses and curricula, instructional materials, assessments, and instructional approaches for laboratory learning. Many of these efforts, particularly those for which a faculty member has been awarded external funding, have a concomitant evaluation and assessment to measure intended student outcomes.

Examined and analyzed as a whole, the empirical knowledge base emerging from funded efforts to improve undergraduate laboratory learning should offer guidance about approaches and interventions that are most promising in effecting positive student outcomes, as well as what kinds of student outcomes can result from different kinds of interventions. The present study examines chemistry laboratory education projects recently funded by the U.S. National Science Foundation (NSF) to describe the "state of the art" that is being identified from the projects' interventions and findings.

Background and literature review

Learning in the laboratory has long had a place in university science education, with a history dating back 200 years in chemistry (Reid & Shah, 2007). As the opening paper (Hofstein & Mamlok-Naaman, 2007) of a special issue of Chemistry Education Research and Practice devoted to laboratory learning in chemistry points out, many researchers have conjectured about its benefits and have studied its outcomes (Hofstein & Lunetta, 1982, 2004; Tobin, 1990; Lazarowitz & Tamir, 1994; Hofstein, 2004; Lunetta et al., 2007). In addressing the purpose of laboratory work in the undergraduate chemistry curriculum, Reid and Shah (2007) point out that there are important tensions to navigate: students learn more when they are engaged and see the relevance of what they are learning, but they may not realize the importance of a particular activity until later in their studies; it is often difficult, however, for instructors to balance the goals of meeting students' needs with communicating their subject. More recently, in an ACS Award address, and later in a published paper, Abraham (2011) presented a review of three decades of research on laboratory learning. The main findings from this review are: 1) while general chemistry instructors identify concepts as the most important outcome of laboratory learning, the instructional strategies that are used do not align with what is known about how to teach these concepts effectively, 2) students are capable of identifying important differences between verification type laboratory experiments and more inquiry-based approaches, including whether there is an emphasis on laboratory skills and scientific processes, and 3) inquiry-based approaches in the laboratory are more effective in promoting both students' conceptual understanding and positive attitudes toward science. This latter conclusion is further elaborated in a study by Emenike et al. (2011), which demonstrated that how laboratory experiments are structured affects how students view their learning. In particular, the structure of a lab experiment as classical (verification), discovery-focused (a version of inquiry-based), or focused on understanding the instrument influenced whether cognitive, affective, or psychomotor prior knowledge was seen by the students to be relevant in their own learning.

What laboratory learning actually comprises is not precisely defined, but rather rests in the intentions of the designer of the instructional materials, the availability of resources and other conditions, the role of the instructor as the facilitator of the laboratory learning experience, and the role of the student as the consumer and interpreter of the experience. An answer to the question of whether laboratory learning achieves its intended outcomes depends not only on the experience, but also on the methods of measuring of those outcomes. Thus, the "state of the art" of laboratory learning has many contingencies.

What Faculty Deem Important

Student outcomes that are intended and measured as part of grant-funded research are related to what faculty deem important as goals for laboratory learning. A recent study found identifiable differences between what faculty cite as the goals of laboratory learning, depending on whether the faculty member is currently or has been engaged in externally funded efforts to improve laboratory learning (Bruck, Towns & Bretz, 2010). Bruck et al. interviewed 22 faculty from two-year colleges and four-year colleges, and found that all chemistry faculty who participated in their study shared two goals for undergraduate laboratory learning: mastery of laboratory techniques and skills, and the development of critical thinking skills and experimental design. However, the authors also found that faculty who had received funding differed from faculty who had not received funding (called "regular faculty") in several ways. For general chemistry, regular faculty cited teamwork skills as an important goal, while grant-funded faculty instead cited connecting lecture and lab. For organic chemistry, both groups of faculty were in agreement on three goals: techniques and lab skills, critical thinking skills, and written communication skills. However, for upper division courses (e.g., analytical chemistry, physical chemistry), there was no common ground: regular faculty focused on laboratory techniques and skills, while grant-funded faculty focused on experimental design and uncertainty in measurement.

Table 1 presents a comparison of the goals for laboratory learning from the two studies.

The Laboratory Activity: What is Inquiry?

Congruent with Abraham's findings (2011), there is general consensus that laboratory activities that engage students in some inquiry-based activity are more conducive to learning than more traditional verification-type laboratory activities. Hofstein et al. (2005) found that students exposed to more inquiry-based laboratory learning were more capable of asking more and better questions for future laboratory investigations. The views of students about the purpose of laboratory are connected to how learning in the laboratory is structured. In a study in which 13 students were interviewed at length about what they viewed to be the purpose of undergraduate laboratory learning experiences in chemistry and their relation to lecture, Russell and Weaver (2008) found that students did not view there to be a connection, particularly to understanding the theory presented in lecture, but rather they perceived that the purpose of the laboratory was to complete the procedures. This conclusion contributed to the premise of continued work by these researchers: "students often fail to make the technical gains that should be possible during their laboratory work... due to a disconnect between students and instructors in terms of laboratory goals and purposes" (Russell & Weaver, 2011, p. 57). In their continued work on this problem, Russell and Weaver (2011) studied student outcomes associated with understanding the nature of science — "what science is, how science is done, and who does it" (p. 58) — when students were engaged in traditional (verification, following established procedures) vs. inquiry-based (students invent questions and design experiments) vs. research-based (integration of authentic science practice, students contribute to generating new scientific knowledge) laboratory curricula. In this study across courses at five universities, they found that students in the research-based labs developed the deepest understanding of the nature of science, and students in the traditional labs developed the shallowest understanding.

There is a long history of debate on what "inquiry" means. Russell and Weaver took a definition of inquiry according to Abraham and Pavelich (1999) in their study, but measures of how inquiry-based a laboratory activity or course is are important if comparisons are to be made across studies. In response to the need to make comparisons among different interventions in undergraduate science laboratory courses, Bruck et al. (2008) developed a rubric to classify the level of inquiry based on characteristics of the laboratory activity and how much independence students have to make decisions about what to do.

Towns and Kraft (2011) recently conducted a review of ten years of empirical research on teaching and learning in chemistry. They included 29 research studies that focused on the laboratory environment. In the review, they conclude that the "body of research doesn't point towards one incontrovertible approach to laboratory" (p. 9). However, they call out two structural approaches to laboratory activities that have been shown to result in positive student outcomes: problem-based learning (PBL) formats, and supporting and developing inquiry skills in the laboratory.

Which Outcomes Are Measured

The desire to implement a pedagogical intervention is always preceded by a belief that its implementation will result in positive student outcomes. If the belief can be unpacked, then it informs the determination of outcomes that should be affected by the intervention. For example, if one believes that a critical feature of being a scientist is to be able to design questions that have the potential to advance society in responsible ways, then an intervention might be designed that incorporates values for social responsibility and opportunities to strengthen the ability to pose questions, and outcomes to be measured might include the extent to which the desire to pursue socially responsible science is sharpened (affective), and the ability to ask relevant questions is enhanced (cognitive). The act of developing a proposal for funding to improve laboratory learning requires articulation of a set of purposes for the laboratory, and these reflect assumptions about what is important in laboratory learning.

Reid and Shah (2007) provide an historical review of the aims of laboratory learning in undergraduate chemistry, and synthesize the body of literature to offer a set of four aims for laboratory work:

]]> • Skills relating to learning chemistry: Making chemistry real, illustrating ideas and concepts, exposing theoretical ideas to empirical testing, and teaching new chemistry.

• Practical skills: Learning to handle equipment and chemicals, practicing safety procedures, mastering specific techniques, measuring accurately, observing carefully.

• Scientific skills: Learning how to observe, deduce, and interpret, appreciating the place of empirical study as a source of evidence, learning how to design experiments that offer genuine insight into chemical phenomena.

• General skills: Working collaboratively, reporting, presenting, discussing, managing time, approaching problems.

They point out that not all of these can be achieved easily by experiments that are typically included in undergraduate chemistry laboratory courses or components. For example, the idea of confirming theory was seen strongly in 19^th century chemistry education and is still very present in undergraduate chemistry education today, but it stands in contrast to exposing theoretical ideas to empirical testing.

There is a variety of ways of organizing types of student outcomes that can result from an effort that deliberately targets specific purposes for the laboratory. One approach to examining student outcomes is to recognize that they must exist within the larger scope of what matters in laboratory learning, and then measure broadly and examine what changes occur. Montagut et al., (2002) have devised an organizational scheme for measuring learning in the chemistry laboratory. It includes five aspects: 1) the quality and appropriateness of the instructional materials, 2) the performance of the instructor, 3) the attitudes and motivation of the student, 4) the competencies achieved by the student in conducting laboratory work, and 5) the quality of the work produced by the student. However, when measuring broadly, there is the danger of missing observing a phenomenon whose grain size is small.

When measuring specifically, there is the attendant difficulty that the act of measuring may cause learning that may not otherwise have occurred. To wit, students learn what the instructor deems valuable to learn by what the instructor assesses. Ramírez et al. (2010) identify this circular argument, and assert that assessments can act to promote particular competencies. In particular, in analyzing basic assessments for seven undergraduate chemistry courses at one university in Argentina, these researchers found that when assessment activities were classified into three categories — 1) memorization and calculation, 2) management of theories and concepts, and 3) integration of conceptual, methodological, and information management — most (77%) of activities fell under the second category, the remainder fell in the first category, and none fell in the third. A similar study in biology by Momsen et al. (2010) analyzed assessments used in 77 introductory biology courses at 44 higher education institutions in the U.S. Through a Bloom's taxonomy analysis of all assessment questions, they found that the vast majority of assessment items (93%) target the lowest two of Bloom's six cognitive levels, and that this holds true regardless of the cognitive level of the course or the type of institution.

How Students Receive Laboratory Learning

In addition to cognitive outcomes of laboratory, there are often affective (i.e., emotion-related) outcomes. These refer, generally, to fostering affective responses through the laboratory that will spill over to the course, to the discipline, or to science more generally. There are three affective outcomes that are frequently identified in the field: attitude, interest, and motivation.

Attitudes. Attitudes are the positive or negative evaluations of an object and the related beliefs about that object. The "object" can be an idea (e.g., science), a process (e.g., experimentation), or a specific item (e.g., the science textbook). The object in this case is usually personally relevant, and the individual experiences some sort of positive emotional experience — this experience is not necessarily enjoyment or entertainment, per se, but can be positive through intellectual challenge. Attitudes are relatively stable, but can be changed through interactions with other people or other objects, especially through an emotional stimulus (McGuire, Lindzey, & Aronson, 1985). Until the work of Thurstone (1928), many psychologists considered it impractical or impossible to measure attitudes reliably. More recent advances in psychometrics and assessment have advanced this field further (cf. Andrich, 1978; Roberts, Laughlin, & Wedell, 1999). In STEM education research, "attitude" is often used to relate to the individual's perception of the discipline (e.g., chemistry) or the field (e.g., science; Reid, 2011).

Motivation. Motivation is a desire to continue with an activity, and is typically related to behaviors that derive, at least in part, from an individual's interest in the object itself or in outcomes that derive from the interaction with the object (Weiner, 2010). Motivation research has distinguished two aspects of motivation: intrinsic and extrinsic. Intrinsic motivation is based on one's interests in a topic or to achieve enjoyment, whereas extrinsic motivation pertains to achieving outcomes (cf. Vallerand & Ratelle, 2004). Compared to interest, motivation is more context dependent, meaning that it is dependent on the individuals' interaction with the object or process for a specific time and in a specific place. Motivation is typically inferred from behavior, such as choice of tasks, effort expended while engaged in such tasks, persistence in these tasks, and associated verbal comments (Pintrich & Schunk, 2002).

How Outcomes Are Measured

A variety of instruments exist for measuring outcomes specific to chemistry, and some are even more specific to laboratory learning in chemistry. In particular, there are two papers that provide overarching principles about evaluating student outcomes in undergraduate chemistry courses. For chemistry courses in which one wishes to evaluate learning in a comprehensive manner, Viera et al. (2007) present dimensions for evaluation based on a constructivist paradigm. For more specific questions probing why particular outcomes occur, a collection of chemical education researchers in the U.S. recently collaborated to provide some guiding principles about assessment of student outcomes in chemistry, and to profile several instruments that can be used to assets the outcomes of curricular reform in chemistry (Holme et al., 2010). The authors argue that, used in conjunction and within the bounds of their validity, these instruments are capable of measuring student problem solving, metacognition, and cognitive development, as well as affective aspects of learning.

Objective of this study

A large fraction of the research and evaluation of innovative approaches to laboratory learning has been funded by public funding agencies. The present study includes an examination of all awards in six programs related to undergraduate chemistry laboratory learning that were awarded from 2000 through 2008 by the National Science Foundation, the federal agency in the U.S. that funds the vast majority of course, curriculum, and laboratory improvements and innovations at the undergraduate level. The research questions under study are: 1) What types of interventions are being funded? 2) What are the intended student outcomes? 3) What are the effects of the interventions on student outcomes? This paper reports on the first two questions for all projects initially funded between 2000 and 2008, and concluded by 2011.

Methods

Data Sources

The original search netted a pool of 99 expired awards. Eliminating duplicates, which occur when separate awards are made to different institutions collaborating on the same project, the number of unique projects was 91. The abstracts of these awards were then read and coded to identify those that focused on undergraduate laboratory learning in chemistry. Projects were excluded if they did not focus explicitly on student learning in the chemistry laboratory. Out of the initial pool of 91 unique projects, 57 projects (or 63%) were found that focused on laboratory learning in chemistry.

Coding Scheme

For each of the 57 undergraduate chemistry laboratory projects, the award documents and any available publications were reviewed and coded. Additionally, other public documents (e.g., Ph.D. theses, conference presentations) were reviewed and coded, if available. All of the documents were coded using an open-coding strategy (Patton, 2002) to capture the variety of interventions and the intended student outcomes. The coding focused on the following areas: the nature of the intervention; the project's focal audience and/or courses; the student learning outcomes identified by the PIs; and, where possible, the project's findings about the effects of the interventions on student outcomes. The coding scheme is summarized in Table 2.

Type of Intervention. Determination of the type of intervention relied primarily on the answer to the question "for whom?" If the intervention involved the development of instructional materials for students in a course or a program, then the project was coded as IM-exp (see Table 2). If the focus was, instead, to develop a curricular approach that could be (or was being) tested in multiple venues, then the project was coded as Curric. Examples of curricular approaches were the development of The Molecules of Life curriculum (Jordan & Kallenbach, n.d.) that integrates interactive pedagogies, web-based molecular modeling exercises, and inquiry-based laboratory experiments, and a focus on using the growth of sprouts as an organizing feature of the entire chemistry curriculum. Instructional interventions (coded Instruc) included developing and testing pedagogical approaches in the laboratory, such as following the Science Writing Heuristic approach. Resource development included the development of remote instrumentation networks and virtual labs. A final category of intervention was a research study centered on how instruction occurs in the lab.

Courses and Audience. Interventions were targeted either at particular courses (e.g., general chemistry for science majors, analytical chemistry), at a set of courses (e.g., bioanalytical chemistry, toxicology, and environmental chemistry), across the entire undergraduate chemistry curriculum, or across two or more disciplines (e.g., chemistry and art history, chemistry and plant biology). While most interventions targeted undergraduate students (UGs) as the primary target audience for the intervention, some also targeted graduate student teaching assistants (TAs) and/or faculty (Fac).

The four aims of undergraduate chemistry laboratory learning of Reid and Shah (2007) were encompassed in two of the three sets of outcomes coded in this study. In particular, 'skills relating to learning chemistry' were collected under cognitive outcomes of the laboratory, while the other three sets of skills are represented under a broader set of skill outcomes. A third set of outcomes recognizes the affective domain.

Cognitive Outcomes. The grouping of cognitive outcomes was informed by the study of Momsen et al. (2010) to code undergraduate biology assessments according to Bloom's taxonomy (Bloom, 1956). Coding in the present study used a rougher level grouping, as assessment items generally were not provided in the documents that were examined, so determination of cognitive outcomes was made based on descriptions of anticipated student outcomes. COG1 encompasses the first two Bloom taxonomy levels, knowing and comprehending. Examples of this include being able to explain how a mass spectrometer works, assigning peaks in an NMR spectrum to particular features of molecules, naming molecules by IUPAC conventions, and relating a 3D molecular structure to its Fisher projection. COG2 encompasses the middle two Bloom taxonomy levels, analyzing and applying. Examples of this include analyzing a data set that is provided, interpreting an NMR spectrum and figuring out whether a compound belongs to a particular class of structures, or determining the rate of a reaction from an energy profile graph. COG3 encompasses the highest two Bloom taxonomy levels, evaluating and creating new knowledge or approaches. Examples of this involve applying what has been learned to novel situations, rather than repeating what was done in the laboratory, for example, designing an experiment to determine something that was not investigated in the laboratory, or critiquing the analysis in a research paper.

Skill and Action Outcomes. Codes initially created mirrored the list of all of the specific laboratory skills identified by Reid and Shah (2007), and then were collapsed into collective codes when tendencies were noticed across projects to group related skills. The remaining collective codes that described all projects that included skills as intended aims were: involvement in (or deepening of) undergraduate research, scientific writing ability, technique in using instruments (e.g., GC/MS, NMR), spatial or visualization ability, and teamwork.

Coding Process

Initially, the public abstracts of the projects were coded, to determine what the intervention was, and to assess, if possible, the intended student outcomes. Based on this initial coding, projects were grouped into the five types of interventions. Other documents for each project were then examined to verify the intervention type. In a few cases, the classification of type of intervention was modified. For example, several projects that were initially classified as instructional materials development were reclassified as curricular approaches when it was recognized that the project was designed around a particular approach to laboratory learning, such as integrating laboratory and lecture, and the assessment of the project involved looking across the entire intervention to determine the extent and quality of this approach.

The materials on each project were then searched to identify intended student outcomes and methods for their measurement. In some cases, assessment instruments were provided in appendices, and in other cases they were described. Some projects reported using published instruments and provided citations. In many cases, documents described the process by which assessments were developed and the expertise of the individuals who contributed to the development, so it was possible to deduce the intended student outcomes that were measured.

Information about every project was sought in publicly available journal articles and other published work. For all 57 projects, searches were run in four journals that are the most common publishing venues of such projects (Journal of Chemical Education, Chemistry Education Research and Practice, Journal of College Science Teaching, and Council on Undergraduate Research Quarterly) to locate published work by the PIs that reported on the projects. NSF's project report template requires including citations of journal articles, which are subsequently automatically drawn from the project reports and appended to the public abstracts available on the NSF award database. Where this occurred, these papers were also included in analysis. The search for papers also included examining the publication sections of CVs of PIs that were available on many faculty members' own websites. In some cases, this resulted in locating other publications that reported on the work of the projects. For example, for one Type 2 project in which the evaluator was a PI, the Ph.D. thesis of a student in the research group that PI was publicly available (Jiang, 2008), making it possible to learn much more about the intended student outcomes of the project and the methods used to measure them. For journal articles found in the four journals searched, a determination was made about whether the paper was related to the NSF project by application of three criteria: 1) if the date of publication was at least six months after the award was made, 2) if the specific NSF project was acknowledged in the paper (this occurred in nearly all cases), 3) in cases where the specific NSF project was not acknowledged, if the paper was about the intervention described in the NSF abstract of the project. The vast majority of published papers were presentations of new laboratory experiments that had been developed under the NSF funding. Some of these papers also included a presentation of evidence that the laboratory activity resulted in specific student outcomes. The journal articles were coded for two main purposes: 1) to verify the intervention type identified in the coding of the NSF documents, and 2) to verify and provide more specific data for coding the cognitive, affective, and skills outcomes studied by the project.

Most projects were focused on studying outcomes for undergraduate students in the laboratory, so codes were identified as 1 (present) or 0 (absent). A few projects were focused on enhancing the abilities of faculty or TAs in teaching the laboratory, and one project focused on studying the goals of faculty for laboratory learning. Codes for student outcomes intended were marked N/A in these cases. Both authors independently coded all projects, and then discussed results to resolve any minor discrepancies that arose. No major discrepancies occurred.

Results

The data were first examined according to the courses in which the interventions took place. Table 3 presents a summary of the data by courses, contrasted with the findings of Bruck et al. (2010), who interviewed a number of chemistry faculty who engage in these innovations. Across all course types, many projects focused on COG 1 student outcomes. By contrast, relatively lower percentages of projects focused on COG 3 student outcomes, with the exception of general chemistry projects (29%). Regarding affective outcomes, a higher proportion of projects that targeted multiple courses across the chemistry curriculum focused on attitudes (70%).

The data were also examined according to the interventions that the projects implemented. Table 4 presents the number of projects for the respective intervention type and the project's intended outcomes (cognitive, affective, and skills). The majority of projects, 47, were Type 1 (IM-exp) interventions, so this will be the focus of analysis. Of the 47 IM-exp interventions, 37 (or 79%) focused on Level 1 cognitive skills, 13 (28%) focused on Level 2 cognitive skills, and just 3 (6%) focused on Level 3 cognitive skills. Regarding affective outcomes, 22 (or 47%) focused on attitudes, 14 (30%) addressed interest, and just 5 (11%) focused on motivation. Finally, regarding skill outcomes, 13 (28%) focused on undergraduate research skills, 14 (30%) focused on laboratory technique outcomes, 6 (13%) focused on students' teamwork abilities, and just 2 (4%) addressed writing skills.

Discussion

This study offers a review and analysis of a detailed facet of the larger body of work in chemical education. At present, there is a National Research Council consensus study in progress on Discipline-Based Education Research (DBER). The charge of the DBER study includes assessing the "status, contributions, and future direction of discipline-based education research (DBER) in physics, biological sciences, geosciences, and chemistry" (Board on Science Education, 2011). In the consensus study, DBER is defined as combining "knowledge of teaching and learning with deep knowledge of discipline-specific science content. It describes the discipline-specific difficulties learners face and the specialized intellectual and instructional resources that can facilitate student understanding."

Reporting on Studies Devoted to Laboratory Learning in Undergraduate Chemistry

During its 30-month duration, the DBER study commissioned several papers on the state of research in the various disciplinary areas of undergraduate science education included in the study. One of these papers was charged with reviewing and summarizing the current state of research on teaching and learning in chemistry (Towns & Kraft, 2011). This paper included in its review 379 peer-reviewed research papers in chemical education published across nine journals. The studies were organized into the following categories (number of papers in parentheses): pedagogy (161), misconceptions (70), particulate nature of matter (45), instrument development (13), student achievement (18), and miscellaneous (72). Within the pedagogies category, 29 of the 161 papers focused on the laboratory environment. There were also a few research papers focused on the laboratory environment in the particulate nature of matter and instrument development categories. As noted in the Introduction of this paper, with regard to the papers on laboratory learning, the authors conclude that "the body of research doesn't point towards one incontrovertible approach to laboratory" (p. 9).

Well over half (63%) of the NSF grants whose projects formed the data in the study presented in this paper were focused on laboratory learning, yet according to the review by Towns and Kraft, less than 10% of the research literature in chemical education is focused on laboratory learning. There are several reasons for this difference; two are discussed here. First, the commissioned paper from the DBER study included research published by authors from many countries, while the data in this study are from the U.S. only. Second, all of the IM-exp projects in this study included an evaluation effort, but most evaluation efforts were small and focused on accountability to the goals of the project and formative evaluation to provide feedback to the PIs on the progress of their work. When only a small part of the project is focused on studying student outcomes, it may not yield sufficient results to produce a research paper. Rather, peer-reviewed publications from these types of projects are usually presentations of novel laboratory experiments and approaches. These publications sometimes present findings about student outcomes as evidence of the value of the laboratory activities, but these findings are not the prime focus of the papers (see Alignment section below). These types of papers were not included in the review paper commissioned by the DBER study. Thus, the present study offers a more detailed review of one facet of the wider body of literature in chemical education, undergraduate laboratory learning, and provides a complement to the paper commissioned by the DBER study.

Faculty Goals for Laboratory Learning in Chemistry

The present study validates some of the findings and offers nuances to other findings of Bruck et al. (2010). As described in the Introduction, Bruck et al. found that mastery of laboratory techniques and skills, and the development of critical thinking skills and experimental design, were the primary goals for laboratory learning deemed important by all faculty who were interviewed. As the present study only examines projects that were funded by NSF, it can be compared to the findings of Bruck et al. about the goals for laboratory learning espoused by grant-funded faculty. Table 3 presents a comparison between grant-funded faculty goals identified in the study by Bruck et al., and goals found to be intended in the funded projects examined in the present study. Several inferences may be drawn from these data.

Very few projects focused on general chemistry target involvement in undergraduate research as an explicit outcome. This is surely due in part to the way in which the data were organized in this study. General chemistry serves all science majors (not just chemistry majors), and some projects included in the general chemistry bin in Table 3 were focused on general chemistry for non-science majors, some were on general chemistry for science majors only, and some focused on both types of courses. Other projects funded by NSF, which were not included in this review, have indeed focused on connecting undergraduate research to general chemistry (cf., Russell & Weaver, 2011; Weaver et al., 2006; Russell et al., 2009). However, it appears that targeting undergraduate research in connection with first-year/general chemistry may be an area for opportunity to learn more that could be valuable in advancing understanding of chemical education.

Mastery of technique in using instruments was seen to be most prominent in upper division courses, and least prominent in general chemistry. Bruck et al. (2010) found that grant-funded faculty tended to place a focus on lab skills and techniques at organic chemistry rather than upper division courses. This discrepancy may reflect a difference in definition of the term "technique" between these studies. What Bruck et al. term "technique" reflects the ability to follow detailed procedures and handle laboratory glassware and other equipment carefully. What was meant by technique in the present study was the skill of operating instruments, such as NMR and FT/IR, properly. These are components of understanding the nature of science, as operationalized by Russell and Weaver (2011), who found that students engaging in more research-oriented laboratory activities at the first-year university level resulted in deeper understanding of the nature of science. Nevertheless, few projects focused explicitly on developing students' skills in experimental design and understanding uncertainty in measurement, so these codes were not retained in the final results. This finding demonstrates a difference between the present study and a conclusion by Bruck et al. (2010) that grant-funded faculty tend to value experimental design and uncertainty in measurement as important goals for upper division courses. This disparity may be due in part to two differences in the data sets. First, the studies had slightly different samples, in that the present study drew only upon information on NSF-funded projects, whereas Bruck et al. (2010) both externally funded and non-externally funded work. Second, the studies used different data collection methods. Data for Bruck et al. (2010) were obtained from interviews of faculty. The present study drew upon data on NSF-funded projects, and publications about what was accomplished.

Purposes of Laboratory

Both America's Lab Report (NRC, 2005) and Reid and Shah (2007) draw primary conclusions that include the need for greater coherence between laboratory learning and lecture. Reid and Shah state that,

Laboratory work in higher education cannot be seen in isolation. For most students it ... has to relate to material taught in lectures and tutorials. However, of greater importance is the need to see the 'hands-on' laboratory time as part of a wider process of learning. (p. 172).

The Committee that produced America's Lab Report based its conclusions on a wide review of research on high school (grades 9-12) laboratory learning, but the Committee's findings may also hold relevance to university laboratory learning:

Conclusion 2: Four principles of instructional design can help laboratory experiences achieve their intended learning goals if: (1) they are designed with clear learning outcomes in mind, (2) they are thoughtfully sequenced into the flow of classroom science instruction, (3) they are designed to integrate learning of science content with learning about the processes of science, and (4) they incorporate ongoing student reflection and discussion. (NRC, 2005, p. 6).

How Laboratory is Anticipated to Change Students

As summarized above, there are often desired affective, cognitive, and skill outcomes for science education (Reid, 2011). These should be seen prominently in the types of interventions that have been studied most. Thus, a comparison is made here between the outcomes studied in the projects included in this study and those observed to have been substantial in the review conducted by Towns and Kraft (2011). In their review of 10 years of research literature in chemical education, Towns and Kraft identified two promising approaches to laboratory learning. First, they identified that problem-based learning has been shown to have a positive effect on students' attitudes and perceptions in the general chemistry and organic chemistry laboratory. Second, they identified across the literature that when pedagogical structures support students' development of inquiry skills in the laboratory, students are better able to ask questions, hypothesize, and suggest good questions for further study.

The present study echoes these findings to some extent, in terms of anticipated affective outcomes. Of the 47 projects focused on instructional materials development, eight (17%) specifically included problem-based learning as a strategy. Among this subset, there was a range of affective outcomes anticipated. Six intended to study changes in attitudes toward science, chemistry and/or the laboratory course, two examined interests (one in lab, and the other in solving problems), and one was interested in motivation to engage in laboratory learning.

A number of projects (13, or 23%) also expected to foster greater engagement by students in undergraduate research. Projects that included an explicit focus on undergraduate research were 2.2 times as likely as projects not explicitly focused on undergraduate research to express a goal that students' technical expertise in using instruments in the laboratory would increase. There were no appreciable relationships between other skills (writing, teamwork, and visualization ability) and undergraduate research as a focus, nor were there any appreciable associations between cognitive or affective outcomes anticipated and whether undergraduate research was a focus. One interpretation of these results is that there was an expectation that, if students use instruments more, they will be more likely to choose to engage in undergraduate research. Another interpretation is that institutions that acquired instruments (e.g., NMR, GC/MS) did so because they desired to provide more opportunities for undergraduate students to engage in research.

Alignment between Goals of the Intervention and Anticipated Outcomes

The projects that were funded represent the best proposals submitted to NSF, as judged through the peer-review process and subsequent decisions by program officers, who take the peer reviews under advisement along with other factors when determining which projects to recommend for funding. The program solicitations (sometimes called program announcements) that called for proposals defined the expectations for what the NSF was interested in funding and how to make a case for why a particular idea should be funded. All versions of the solicitation for this program have in common the following information regarding the use of and contribution to knowledge about science, technology, engineering and mathematics (STEM) education, and expectations for measurable outcomes (NSF, 2011):

Use of and Contribution to Knowledge about STEM Education: Projects should reflect high quality science, technology, engineering, and mathematics. They should have a clear and compelling rationale, use methods derived from existing knowledge concerning undergraduate STEM education, build on existing projects of a similar nature, and present evidence supporting the approach. They also should have an effective approach for adding to this knowledge by disseminating their results.

]]> ...

Expected Measurable Outcomes: Projects should have goals and objectives that have been translated into a set of expected measurable outcomes that can be monitored using quantitative or qualitative approaches or both. These outcomes should be used to track progress, guide the project, and evaluate its ultimate success. Some of the expected measurable outcomes should pay particular attention to student learning, contributions to the knowledge base, and community building.

These communicate an expectation that, to be successful, proposed interventions should contain an alignment between the need for an intervention, a rationale based on evidence for why a proposed approach should be successful in achieving an expected set of outcomes, and a method for determining whether the outcomes are achieved. Most undergraduate chemistry laboratory projects that have been completed under this funding program (47 out of 57 during 2000-2010) have had primary goals related to developing new instructional materials, curriculum, or approaches to doing experiments in the laboratory. However, they also have all included measurable outcomes related to student learning. Additionally, all projects are expected to contribute to the knowledge base in STEM education. Formal contributions to the literature are, of course, not the only way to contribute to the knowledge base. Emerging from the 57 projects examined in this study (some of which concluded only recently, so it is likely that there are publications in preparation or in press), so far there have been 22 papers published in peer-reviewed journals: 14 of the 22 were papers on course development efforts or laboratory activities (Brown, 2007; Birdwhistell et al., 2008; Cancilla & Albon, 2008; Zovinka & Stock, 2010; Durham et al, 2011; Faraldos et al., 2011; Schurter et al., 2011; Szalay et al., 2011; Lanigan, 2008; deProphetis et al., 2010; Jin & Bierma, 2010; Nivens et al., 2010; Stanford, 2011; VanDorn et al., 2011), with the last 5 of these having reported an evaluation of student outcomes within the paper; and 7 of the 22 were research studies related to how students learn effectively in the laboratory (Fay et al., 2007; Leinhardt et al., 2007; Rudd et al., 2007; Evans et al., 2008; Schroeder & Greenbowe, 2008; Bruck et al., 2010; Jiang et al., 2010; Rauschenberger & Sweeder, 2010). A theory-based rationale for why an intervention should work can inform what the expected outcomes should be, and can also inform what to measure so that a mechanism can be uncovered for how and why the outcomes occur or not. Future study could investigate the relationships among laboratory activity projects' theory of action (the reason an intervention should work), theory of learning (what the investigators believe to be true about learning), and the resulting student learning outcomes.

Conclusions

An analysis was conducted on all of the projects related to undergraduate chemistry learning that were funded by the NSF's Course, Curriculum and Laboratory Improvement program and concluded between 2000 and 2011. Of 91 unique projects, 57 focused on laboratory learning. Of these, the vast majority (47) were interventions that involved developing new instructional materials for undergraduate chemistry laboratory activities. Assessment of student outcomes generally included both cognitive and affective outcomes, and often also included skills. Approaches employed in many interventions are well aligned with the conclusions of a consensus study of the National Academies of Science and of researchers studying the purposes of laboratory learning in undergraduate science courses.

Bibliography

Abraham, M. R. What can be learned from laboratory activities? Revisiting 32 years of research, Journal of Chemical Education, 88, 1020-1025, 2011.         [ Links ]

Abraham, M. R. & Pavelich M. J., Inquiries into chemistry (3rd ed.), Long Grove, IL: Waveland Press, Inc., 1999.         [ Links ]

Andrich, D., Relationships Between the Thurstone and Rasch Approaches to Item Scaling, Applied Psychological Measurement, 2, 451-462, 1978.         [ Links ]

Birdwhistell, K. R., Nguyen, A., Ramos, E. J., & Kobelja, R., Acylation of ferrocene: A Greener approach, Journal of Chemical Education, 85(2), 261-2, 2008.         [ Links ]

Bloom, B. S. Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook I: Cognitive Domain, New York: Longmans, Green, 1956.         [ Links ]

Board on Science Education. Status, Contributions, and Future Direction of Discipline-Based Education Research (DBER), 2011. Retrieved from http://www7.nationalacademies.org/bose/DBER_Homepage.html.         [ Links ]

Brown, B. J., Sprouts on a shoestring, The Science Teachers Bulletin, 70, 16, 2007.         [ Links ]

Bruck, L. B., Towns, M., & Bretz, S. L., Faculty perspectives of undergraduate chemistry laboratory: Goals and obstacles to success, Journal of Chemical Education, 87(12), 1416-1424, 2010.         [ Links ]

Bruck, L. B., Bretz, S. L., & Towns, M. H. Characterizing the level of inquiry in the undergraduate laboratory, Journal of College Science Teaching, 38(1), 52-58, 2008.         [ Links ]

Cancilla, D., & Albon, S., Reflections from the Moving the Laboratory Online Workshops: Emerging Themes, Journal of Asynchronous Learning Networks, 12, 53-59, 2008.         [ Links ]

Carnduff, J., and Reid, N., Enhancing undergraduate chemistry laboratories, pre-laboratory and post-laboratory exercises, examples and advice, Education Department, Royal Society of Chemistry, Burlington House, Piccadilly, London, 2003.         [ Links ]

deProphetis Driscoll, W., Gelabert, M., & Richardson, N. Efficacy of using learning communities to improve core chemistry education and increase student interest and retention in chemistry, Journal of Chemical Education, 87(1), 49-53, 2010.         [ Links ]

Durham, C. R., Chase, J. M., Nivens, D. A., Baird, W. H., & Padgett, C. W. Chemical environmental effects on Kβ/Kα intensity ratio: An X-ray fluorescence experiment on periodic trends, Journal of Chemical Education, 88(6), 819-821, 2011.         [ Links ]

Eminike, M. E., Danielson, N. D., & Bretz, S. L. Meaningful learning in a first-year chemistry laboratory course: Differences across classical, discovery, and instrumental experiments, Journal of College Science Teaching, 41(2), 89-97, 2011.         [ Links ]

Evans, K. L., Yaron, D., & Leinhardt, G. Learning stoichiometry: A comparison of text and multimedia formats, Chemistry Education Research and Practice, 9, 208-219, 2008.         [ Links ]

Faraldos, J. A., Giner, J.-L., Smith, D. H., Wilson, M., Ronhovde, K., Wilson, E., Clevette, D., Holmes, A., & Rouhler, K., Enzymatic resolution of 1-phenylethanol and formation of a diasteromer: An undergraduate 1-H NMR experiment to introduce chiral chemistry, Journal of Chemical Education, 88(3), 334-336, 2011.         [ Links ]

Fay, M. E., Grove, N. P., Towns, M. H., & Bretz, S. L., A rubric to characterize inquiry in the undergraduate chemistry laboratory, Chemical Education Research and Practice, 8, 212-219, 2007.         [ Links ]

Hofstein A., The laboratory in chemistry education: thirty years of experience with developments, implementation and evaluation, Chemistry Education Research and Practice, 5, 247-264, 2004.         [ Links ]

Hofstein A. & Lunetta, V. N., The role of the laboratory in science teaching: neglected aspects of research, Review of Educational Research, 52, 201-217, 1982.         [ Links ]

Hofstein A. & Lunetta V. N., The laboratory in science education: foundation for the 21^st century, Science Education, 88, 28-54, 2004.         [ Links ]

Hofstein, A., Navon, O., Kipnis, M., & Mamlok-Naaman, R., Developing students' ability to ask more and better questions resulting from inquiry-type chemistry laboratories, Journal of Research in Science Teaching, 42(7), 791-806, 2005.         [ Links ]

Hofstein, A., & Mamlok-Naaman, R., The laboratory in science education: The state of the art, Chemistry Education Research and Practice, 8(2), 105-107, 2007.         [ Links ]

Holme, T., Bretz, S. L., Cooper, M., Lewis, J., Paek, P., Pienta, N., Stacy, A., Stevens, R., & Towns, M., Enhancing the role of assessment in curriculum reform in chemistry, Chemical Education Research and Practice, 11, 92-97, 2010.         [ Links ]

Jiang, B., Ph.D. thesis, University of South Florida: Formal reasoning and spatial ability: A step towards "science for all", 2008. Retrieved from http://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1317&context=etd.         [ Links ]

Jiang, B., Xu, X., Garcia, Al, & Lewis, J. E., Comparing two tests of formal reasoning in a college chemistry context, Journal of Chemical Education, 87(12), 1430-1437, 2010.         [ Links ]

Jin, G., & Bierma, T., Guided-inquiry learning in environmental health, Journal of Environmental Health, 73(6), 80-85, 2011.         [ Links ]

Jordan, T., & Kallenbach, N. R. (n.d.). Molecules of Life: A Chemical Approach. Textbook under development.         [ Links ]

Krapp, A., Interest, motivation and learning: An educational-psychological perspective, European Journal of Psychology of Education, 14, 23-40, 1999.         [ Links ]

Krapp, A., An eduational-psychological theory of interest and its relation to SDT. In: E. L. Deci & R. M. Ryan (eds.), Handbook of self-determination research. Rochester, NY: University of Rochester Press, 2004.         [ Links ]

Lanigan, K. C., Teaching analytical method development in an undergraduate instrumental analysis course, Journal of Chemical Education, 85, 138-140, 2008.         [ Links ]

Lazarowitz R. & Tamir P., Research on using laboratory instruction in science. In: D. L. Gabel. (ed.) Handbook of research on science teaching and learning (pp. 94-130), New- York: Macmillan, 1994.         [ Links ]

Leinhardt, G., Cuadros, J., & Yaron, D., "One firm spot": The role of homework as lever in acquiring conceptual and performance competence in college chemistry, Journal of Chemical Education, 84(6), 1047, 2007.         [ Links ]

Lunetta V. N., Hofstein A. and Clough M., Learning and teaching in the school science laboratory: an analysis of research, theory, and practice. In: N. Lederman. and S. Abel (eds.), Handbook of research on science education (pp. 393-441). Mahwah, NJ: Lawrence Erlbaum, 2007.         [ Links ]

McGuire, W., Lindzey, G., & Aronson, E., Attitudes and attitude change. Handbook of social psychology: Special fields and applications, 2, 233–346, 1985.         [ Links ]

Momsen, J. L., Long, T. M., Wyse, S. A., & Ebert-May, D., Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills, CBE-Life Sciences Education, 9, 435-440, 2010.         [ Links ]

Montagut B., P., Sansón O., C., & González M., R. Ma., Evaluación del aprendizaje en situaciones de laboratorio, Educ. quím., 13(3), 188-200, 2002.         [ Links ]

National Research Council. America's Lab Report, National Academies Press: Washington, DC, 2005.         [ Links ]

National Science Foundation (2011). Solicitations 10-544, 09-529, 08-546, 07-543, 06-536, and 05-559 retrieved from http://www.nsf.gov/publications/pub_summ.jsp?WT.z_pims_id=5741&ods_key=nsf10544.         [ Links ]

Nivens, D. A., Padgett, C. W., Chase, J. M., Verges, K. J., & Jamieson, D. S., Art, meet chemistry; Chemistry, meet art: Case studies, current literature, and instrumental methods combined to create a hands-on experience for nonmajors and instrumental analysis students, Journal of Chemical Education, 87(10), 1089-1093, 2010.         [ Links ]

Patton, M. Q. Qualitative Research and Evaluation Methods, 3rd ed. Sage Publications: Thousand Oaks, CA, 2002.         [ Links ]

Pintrich, P. R., & Schunk, D. H. Motivation in education: Theory, research, and applications. Englewood Cliffs, NJ: Merrill, 2002.         [ Links ]

Ramírez, S., Viera, L., & Wainmaier, C., Evaluaciones en cursos universitarios de Química: ¿qué competencias se promueven?, Educ. quím., 21(1), 16-21, 2010.         [ Links ]

Rauschenberger, M. M. & Sweeder, R. D., Gender performance differences in biochemistry, Biochemistry and Molecular Biology Education, 38(6), 380-384 2010.         [ Links ]

Reid, N., Attitude research in science education. In: I. M. Saleh & M. S. Khine (eds.), Attitude research in science education: Classic and contemporary measurements. Charlotte, NC: Information Age Publishing, 2011.         [ Links ]

Reid., N., & Shah, I., The role of laboratory work in university chemistry, Chemistry Education Research and Practice, 8(2), 172-185, 2007.         [ Links ]

Roberts, J. S., Laughlin, J. E., & Wedell, D. H., Validity issues in the Likert and Thurstone Approaches to Attitude Measurement, Educational and Psychological Measurement, 59, 211-233, 1999.         [ Links ]

Rudd, J. A., Greenbowe, T. J., & Hand, B. M., Using the Science Writing Heuristic to improve students' understanding of general equilibrium, Journal of Chemical Education, 84(12), 2007-2011, 2007.         [ Links ]

Russell, C. B., & Weaver, G. C., Student perceptions of the purpose and function of the laboratory in science: A grounded theory study, International Journal for the Scholarship of Teaching and Learning, 2(2), 2008. Retrieved from http://academics.georgiasouthern.edu/ijsotl/v2n2/articles/_Russell/index.htm.         [ Links ]

Russell, C. B., Bentley, A. K., Wink, D. J., & Weaver, G. C., Materials development for a research-based undergraduate laboratory curriculum, The Chemical Educator, 14, 55, 2009.         [ Links ]

Russell, C. B., & Weaver, G. C., A comparative study of traditional, inquiry-based, and research-based laboratory curricula: Impacts on understanding of the nature of science, Chemistry Education Research and Practice, 12(1), 57-67, 2011.         [ Links ]

Schroeder, J. D. & Greenbowe, T. J., Implementing POGIL in the lecture and the Science Writing Heuristic in the laboratory – student perceptions and performance in undergraduate organic chemistry, Chemical Education Research and Practice, 9, 149-156, 2008.         [ Links ]

Schurter, E. J., Zook-Gerdau, L. A., & Szalay, P. S., Analysis of a suspected drug sample, Journal of Chemical Education, 88, 1416-1418, 2011.         [ Links ]

Stanford, D., Successful student transitions from the community college to the four-year college facilitated by undergraduate research, Council on Undergraduate Research, 31(3), 16-22, 2011.         [ Links ]

Szalay, P. S., Zook-Gerdau, L. A., & Schurter, E. J., A multi-technique forensic experiment for a nonscience-major chemistry course, Journal of Chemical Education, 88, 1419-1421, 2011.         [ Links ]

Thurstone, L. L., Attitudes Can Be Measured, American Journal of Sociology, 33, 529-554.         [ Links ]

Tobin K. G., Research on science laboratory activities; in pursuit of better questions and answers to improve learning, School Science and Mathematics, 90, 403-418, 1928, 1990.         [ Links ]

Towns, M. H., & Kraft, A., Review and synthesis of research in Chemical Education from 2000-2010, 2011. Retrieved from http://www7.nationalacademies.org/bose/DBER_Towns_October_Paper.pdf.         [ Links ]

Vallerand, R. J., & Ratelle, C. F. Intrinsic and extrinsic motivation: A hierarchical model. In: E. L. Deci & R. M. Ryan (eds.), Handbook of self-determination research. Rochester, NY: University of Rochester Press, 2004.         [ Links ]

VanDorn, D., Ravalli, M. T., Small, M. M., Hillery, B., & Andreescu, S., Adsorption of arsenic by iron oxide nanoparticles: A versatile, inquiry-based laboratory for a high school or college science course, Journal of Chemical Education, 88(8), 1119-1122, 2011.         [ Links ]

Viera, L., Ramírez, S., Wainmaier, C., & Salinas, J., Criterios y actividades para la evaluación del aprendizaje en cursos universitarios de química, Educ. quím., 18(4), 294-302, 2007.         [ Links ]

Weaver, G. C., Wink, D., Varma-Nelson, P., Lytle, F., Morris, R., Fornes, W., Russell, C., & Boone, W. J., Developing a new model to provide first- and second-year undergraduates with chemistry research experience: Early findings of the Center for Authentic Science Practice in Education (CASPiE), The Chemical Educator, 11, 125, 2006.         [ Links ]

Weiner, B., The Development of an Attribution-Based Theory of Motivation: A History of Ideas, Educational Psychologist, 45, 28-36, 2010.         [ Links ]

Zovinka, E. P., & Stock, A. E., Microwave instruments: Green machines for green chemistry?, Journal of Chemical Education, 87(4), 350-352, 2010.         [ Links ]

Nota

* Disclaimer: This work was conducted in part while the authors were under employment of the National Science Foundation, including support through an Independent Research and Development plan. The findings and opinions expressed are solely those of the authors, and do not necessarily reflect the opinions, positions, or policies of the National Science Foundation.