NandiBERT: An Encoder Balancing English and Indian Languages

NandiBERT series small model released by FrontiersMind on 11 18: The encoder has 1.46 billion parameters (with an unbound MLM output layer of 7700 ten-thousand parameters stored separately in the file), 28 layers, and a hidden dimension of 768. It uses grouped-query attention, QK-norm, and SwiGLU, with a context length of 16,384 tokens and a vocabulary of 10 ten-thousand words using rank-192 factorized embeddings. It covers English and 13 Indian languages in their native scripts (with 11 of them also supporting Romanized transliteration), as well as Southeast Asian and Middle Eastern languages. It is a pre-trained encoder intended for downstream fine-tuning, targeting classification, named entity recognition, reranking, and retrieval.

Original title: FrontiersMind/NandiBERT-150M

NandiBERT series small model released by FrontiersMind on 11 18: The encoder has 1.46 billion parameters (with an unbound MLM output layer of 7700 ten-thousand parameters stored separately in the file), 28 layers, and a hidden dimension of 768. It uses grouped-query attention, QK-norm, and SwiGLU, with a context length of 16,384 tokens and a vocabulary of 10 ten-thousand words using rank-192 factorized embeddings. It covers English and 13 Indian languages in their native scripts (with 11 of them also supporting Romanized transliteration), as well as Southeast Asian and Middle Eastern languages. It is a pre-trained encoder intended for downstream fine-tuning, targeting classification, named entity recognition, reranking, and retrieval.

Why it matters For classification, entity recognition, and retrieval reranking, you no longer need separate encoders for English and Indian languages; officially, it claims top performance in both areas among models of the same size

Upvotes 28 · Likes 31 · Downloads 173

Reasons

  • Official average score on Indian language benchmarks is 67.37, higher than IndicBERT-v2 with 2.72 billion parameters (66.57)
  • Average score on English benchmarks is 62.20, higher than ModernBERT-base (60.69) and Ettin-Encoder-150M (61.48), but lower than the English-only UltraBERT-150M (64.59)
  • The same series also includes NandiBERT-450M with 4.45 billion parameters and 32K context; licensed under Apache-2.0

Usage

  • Use transformers 5.x with trust_remote_code=True, and load with AutoModelForMaskedLM for fill-in-the-blank tasks; use AutoModelForSequenceClassification / AutoModelForTokenClassification for classification and NER, which reuses the pre-trained MLM head
  • Official recommendation for NER is a learning rate of 3e-5; lower learning rates like 5e-6 perform significantly worse; when inputting by word, pass the original words directly

Risks

  • It is an encoder requiring fine-tuning and cannot engage in dialogue or generate text; the technical report has not been released, training data is not disclosed, and Chinese is not among the supported languages
  • Scores on IndicQA and MASSIVE slot filling are lower than IndicBERT-v2; results are self-reported by the official team (averaged over three random seeds) and have not been independently verified in this entry

Open

Hugging Face Trending

Explore the latest AI Radar updates