Uncovering Challenges with Assessment in the California Dual Language Learner Pilot Study

In the fall of 2017, the Dual Language Learner Pilot Study launched in California to identify educational practices that help young dual language learners (DLLs) thrive. The DLL Pilot Study is led by experts at the American Institutes for Research and funded at $20 million by the First 5 California Commission. Approximately 60 percent of children under the age of five in California are DLLs and they speak over 65 languages, the most common being Spanish, Mandarin, Cantonese, and Vietnamese. The DLL Pilot Study examines the teaching practices, professional development, and family engagement strategies that promote the development of this population of young children in 16 counties across the state.

One major barrier that researchers have repeatedly encountered during the study has been the lack of assessment tools available in languages other than English that accurately measure children’s academic progress. This inspired the release of the February 2021 report, Challenges in Assessing California’s Diverse Dual Language Learners. The report details researchers’ assessment methods and provides recommendations for practitioners, researchers, and policymakers to better serve DLLs moving forward.

According to the report, researchers in the study varied use of assessments by children’s age and home language, choosing to limit their study to the four languages most commonly spoken by DLLs in California. Given the constraints of conducting one-time assessments on infants and toddlers, whose skills are best gauged through ongoing observations, researchers surveyed the children’s parents and educators. Infant and toddler parents and educators were given the MacArthur-Bates Communicative Development Inventories (MB-CDI) which measure children’s use and understanding of gestures, words, and sentences. The MB-CDI provides adapted versions of their English test in nearly 100 languages, including the focal languages of the DLL Pilot Study.

Parents, educators, and researchers used additional assessments to measure children’s communication, motor skills, problem solving, social-emotional development, and math and literacy skills. Unfortunately, of the seven assessments administered, only two—the MB-CDI and the Minnesota Executive Function Scale—were readily available and adapted to English, Spanish, Mandarin, Cantonese, and Vietnamese.

Complications with translating the remaining five assessments were abundant. Some literacy concepts were not translatable between languages, such as phonological awareness, or the ability to understand sounds within words. While phonological awareness is an essential early literacy skill in English and Spanish, Mandarin and Cantonese characters are not tied to phonemes in the same way, rendering that concept void in translated literacy tests. A researcher-created exam that asked children to identify letters and words in a book was also conceptually invalid using Cantonese or Mandarin characters. Therefore, researchers used an English-language book, while prompting children in Cantonese and Mandarin to find letters or words on the page.

Maintaining a relevant cultural context in translated assessments was also challenging. Researchers decided to use the Woodcock-Johnson math and literacy exams in both English and the children’s home language, but the tests were only available in English and Spanish. Since the English and Spanish versions asked different questions, and researchers sought to administer two unique assessments, they translated the Spanish version into Mandarin, Cantonese, and Vietnamese. Translating from Spanish to Asian languages required consideration of the cultural relevance of the concepts presented. For example, the Spanish picture vocabulary test asked children to name a cornucopia, an item unfamiliar to Chinese cultures. The Chinese reviewers recommended the use of a treasure pot instead, which was conceptually similar, but may not have been as linguistically demanding, which was the intention of the question. The Woodcock-Johnson Oral Language Inventory IV is designed to become more challenging as children progress through the assessment, but translating the test made ensuring difficulty of progression nearly impossible.

In the report, researchers provide recommendations to promote better collection of valid, reliable data for DLLs. When translating exams from one language to another, researchers stress the importance of hiring translators who are fluent in both languages, avoiding translations into intermediary languages like English, and conducting back-translations with native speakers to check for conceptual or grammatical errors. Languages also vary regionally, so researchers recommend partnering with regional language experts who can validate the vocabulary in assessment questions and anticipated responses. A car, for instance, could be a carro for children with Colombian or Venezuelan backgrounds, a coche for children with Mexican or Spanish backgrounds, or an auto if children’s families come from Argentina, Uruguay, Peru, or Chile. All of these words should be accepted as valid responses and incorporated into assessments, depending on the intended recipients.

Researchers recommend informing translators about the format of assessments, such as whether questions will be administered to an entire group of children or individually, which can change the grammar and pronouns that are needed. As more emphasis is placed on testing children in their home languages, researchers of the DLL Pilot Study also urge the norming and validation of assessments on DLLs living in the United States who speak the target language to ensure that tests retain their integrity, difficulty, and relevancy. Only when tests have been validated and normed between languages, and the scores have been equated, can children’s progress on these assessments be compared by researchers, practitioners, and policymakers.

Guaranteeing access to assessments that accurately measure DLL children’s abilities is essential so that children’s strengths can be honored and areas of need can be supported. In California, the political will exists to make these assessments available. The 2017 California State Board of Education’s English Learner Roadmap includes “a system of culturally and linguistically valid and reliable assessments” as one of their key principles. Additionally, the Master Plan for Early Learning and Care, released in December 2020, repeatedly mentions the importance of assessment tools designed for DLLs that align with and strengthen instruction. Whether the funding to develop these high-quality assessment tools and systems will become available remains to be seen.

Researchers from the DLL Pilot Study are concluding the final phase of the study by June 2021, and findings will be disseminated soon after. Our colleagues in the English Learner team at New America will be following the outcomes closely and sharing the findings as they become available.

Enjoy what you read? Subscribe to our newsletter to receive updates on what’s new in Education Policy!