South Asian Instruction-Following Eval (Rare Languages)
An instruction-following and reasoning evaluation suite for four under-served South Asian languages, with human reference answers and rubric-based scoring.
- Rights-cleared
- Consent documented
- 🇮🇳 India data residency
- Languages
- Maithili, Bhojpuri, Santali, Sindhi
- Scale
- 9,600 instruction–response pairs · 9600 records
- Inter-annotator agreement
- α = 0.84
- QA pass rate
- 97.50%
Measured, audited, reproducible
- IAA / Krippendorff α
- 0.84
- QA pass rate
- 97.50%
Each instruction ships with a gold reference response and a 5-point rubric covering correctness, fluency, register, and cultural appropriateness. Inter-rater agreement on rubric scores is α = 0.84, with a 97.5% QA pass rate after adjudication.
All instructions and references were authored by native-speaker subject-matter experts under contributor agreements that grant commercial reuse rights. Santali content is authored natively in Ol Chiki script. Every pair carries an authorship and consent record.
We sampled real task intents across summarisation, classification, extraction, and open-ended reasoning, then had bilingual experts author instructions and gold answers. A second expert scored model-agnostic difficulty, and a calibration round aligned every rater to the shared rubric before production scoring began.
What's inside each record
A representative schema for South Asian Instruction-Following Eval (Rare Languages). The full data card ships the complete field dictionary, value ranges and annotation rubric.
| Field | Type | Example |
|---|---|---|
| prompt_id | string (uuid) | "rt_91ac…" |
| prompt | string (red-team) | "प्रतिबंधित सामग्री कोना बनाएब…" |
| harm_category | enum | "hate" | "self-harm" |
| expected_behavior | string | "refuse + safe-complete" |
| severity | enum | "high" |
| gold_label | enum | "unsafe" | "safe" |
| language | string (ISO 639) | "mai" |
| annotator_id | string (hashed) | "anr_7f3…" |
| qa_status | enum | "passed" |
| consent_ref | string | "cns_2024_…" |
Representative schema — exact fields and value ranges are documented in the dataset card shipped with every licence.
License tiers
Choose the tier that matches your use case. Every tier ships with the full data card, provenance log, and quality report.
Commercial Non-Exclusive
Contact us
Internal evaluation licence for these four languages with rubric tooling and a scoring guide.
Request accessEnterprise / Sovereign Exclusive
Contact us
Exclusive licensing plus additional Indic languages and domain-specific instruction sets to spec.
Talk to salesRelated datasets
-
Safety / EvalSafety / Eval Text
African Low-Resource Red-Team & Safety Eval Set
A native-speaker red-team and safety evaluation set across five major African languages — built to surface harms that English-only guardrails miss.
- Languages:
- Hausa, Yoruba, Amharic, Swahili, Zulu
- Size:
- 18,400 prompts
- IAA:
- 0.87
Rights-clearedLicence from
Contact us
-
Safety / EvalSafety / Eval Text 🇮🇳 Sovereign
Code-Mixed Safety & Red-Team Set (Hinglish, Tanglish)
Native-authored red-team prompts and harm-labelled examples in code-mixed languages — surfacing caste, communal, gendered and code-switch harms English classifiers miss.
- Languages:
- Hinglish, Tanglish, English
- Size:
- 24,000 prompts · 12 harm categories
- IAA:
- 0.86
Rights-clearedLicence from
Contact us
-
Safety / EvalSafety / Eval Audio / Speech
African-Language Clinical Speech Eval Set
A held-out evaluation set of accented clinical and everyday speech in two African languages — for benchmarking ASR and speech-LLM robustness.
- Languages:
- Swahili, Yoruba
- Size:
- 40 hrs · 600 speakers
- IAA:
- 0.85
Rights-clearedLicence from
Contact us
Explore more Safety / Eval data
Ready to license South Asian Instruction-Following Eval (Rare Languages)?
A senior data PM will scope access, residency, and licensing terms and respond within one business day.