BULaMU brings tiny code-switching models to low-resource languages
BULaMU creator mwebazarick introduced a family of 20M, 47M, and 110M parameter language models trained from scratch for Luganda. The creator says an expanded 40-billion-token corpus includes Swahili, Lao, Amharic, Somali, Kinyaranda, and English, with East African languages making up nearly 90% of it, and that a 47M-parameter model was fine-tuned for cross-language story generation. Models and a demo are available through Hugging Face, although the creator's claim that the models can run offline on phones without GPUs or internet connections remains unverified.
The downloadable tiny models offer a practical avenue for experimenting with code-switching and text generation in low-resource languages on modest hardware, though their performance has not been independently assessed.
| Hi Everybody, I am just writing to share some progress I have made on my project. I am the creator of BULaMU, a family of language models (20M, 47M, 110M) trained completely from scratch for a low resource language: Luganda. The models are small and compute-efficient enough to run offline on a phone without requiring a GPU or internet connection. Over the last few months, I increased the corpus ten fold (to 40 billion tokens) by including text from 5 other East African Languages (Swahili, Lao, Amharic, Somali, Kinyaranda) and English. East African Languages make up nearly 90% of the new training corpus. I trained a 47M parameter model on this corpus and then fine-tuned it to be able to generate responses (stories) between the different languages. I have attached a link below if you would like to try a demo of it. I also made the models available on Huggingface if you would like to experiment with them on your own. Demo: https://tinyafricanstories.streamlit.app/ Base Model: https://huggingface.co/mwebazarick/BULaMU-3-42M-Base Fine-tuned ("Story-telling") model: https://huggingface.co/mwebazarick/BULaMU-3-42M-Story submitted by /u/AgencyInside407 [link] [comments] |