Written by 
Professor Inseok Hwang from the Department of Computer Science and Engineering, along with students Jungeun Lee, Suwon Yoon, and Kyoosik Lee from the Department of Computer Science and Engineering at POSTECH in collaboration with Professor Dongsun Yim from Ewha Womans University’s Department of Communication Disorders have created an innovative system for generating personalized storybooks. This system utilizes generative artificial intelligence and home IoT technology to assist children in language learning. Their research was showcased at the “ACM CHI (ACM SIGCHI Conference on Human Factors in Computing Systems),” the leading conference in human-computer interaction, where it earned an “Honorable Mention Award,” recognizing it as one of the top 5% of submissions.
Children’s language development is crucial as it impacts their cognitive and academic growth, their interactions with peers, and overall social development. It is essential to regularly evaluate language progress and provide timely language interventions1) to support language acquisition. The issue is that children grow up in diverse environments, leading to variations in their exposure to vocabulary. However, traditional approaches often rely on standardized vocabulary lists and pre-made storybooks or toys for language skill assessments and interventions, lacking the diversity support.
Recognizing the shortcomings of conventional, one-size-fits-all approaches that fail to address the diverse backgrounds of children, the team created an innovative educational system tailored to each child’s unique environment. They began by employing home IoT devices to capture and monitor the language children hear and speak in their daily lives. Through speaker separation2) and morphological analysis techniques3), the researchers examined the vocabulary children were exposed to, the words they spoke, and those they heard but did not vocalize. They then assessed each word by calculating scores for each word based on key factors relevant to speech pathology.
To create personalized educational materials, the team utilized advanced generative AI technologies, including GPT-4 and Stable Diffusion. This enabled them to produce custom children’s books that seamlessly integrate the target vocabulary for each individual child. By combining speech pathology theory with practical expertise, the researchers developed an effective and personalized language learning system.
The researchers designed the system to accommodate variations in children’s language development by allowing for individualized weighting of factors and flexible vocabulary selection criteria. The system can automate both the extraction of target vocabulary for each child and the creation of personalized storybooks, ensuring that both the vocabulary and the storybooks could be continuously updated in response to changes in the child’s language development and environment. After testing the system in 9 families over a four-week period, the results showed that children effectively learned the target vocabulary, demonstrating the system’s applicability in everyday settings beyond the therapy room.
Jungeun Lee from POSTECH, the lead author of the paper, expressed her aspirations by commenting, “We effectively addressed the limitations of traditional, one-size-fits-all approaches to child language assessment and intervention by using generative AI.” She added, “Our goal is to leverage AI to create customized guides tailored to different individuals’ levels and needs.”
Professor Inseok Hwang from POSTECH, the corresponding author, remarked, “Through interdisciplinary research, we have successfully developed a personalized language stimulation and development system that integrates generative AI technology with speech pathology theory.” He continued, “We hope our findings will encourage educators to respect and incorporate the diverse environments and learning goals of children.”
Co-author Professor Dongsun Yim from Ewha Womans University also expressed her expectation by saying, “Our work demonstrates the potential for non-traditional, personalized language support services.” She added, “The system showcases the ability to tailor target vocabulary extraction and linguistic stimuli delivery for children exposed to varied environments and languages.”
The research was conducted with support from the Mid-Career Researcher Program of the National Research Foundation of Korea, the SSK, the ITRC of the IITP, and the ICT R&D Innovation Voucher Program.