Doctoral Researcher investigating unified Multimodal, Multilingual, and Multitask (M3) Language Models (M3LM), bridging a decade of production AI engineering with foundational speech-text representation learning. My research resolves the topological conflict between continuous acoustic paralinguistics and discrete semantic tokens across 100+ global and underrepresented languages, ensuring AI systems serve diverse linguistic identities with demographic fairness, structural equity, and hardware-level efficiency.

On-going Research

Education

Professional Experience

Google Scholar

	M3LM-SpeechEncoder: Multi-View Acoustic and Semantic Fusion for Foundation Models Barathi Ganesh HB , Michal Ptaszynski, Reshma Unnikrishnan, Jairam R Compuational Linguistics [abs] [pdf] [code] Spoken language carries two fundamentally opposing forms of information: discrete linguistic content, which requires abstracting away speaker variance, and continuous paralinguistic identity, which relies directly on vocal-tract resonance and prosody. Contemporary Audio Large Language Models (Audio-LLMs) typically force both modalities through a single homogenized stream or early discrete tokenizers, resulting in phonetic blurring, loss of biometric fidelity, and severe demographic biases under noisy, real-world conditions. To resolve this conflict, we propose the M3LM-SpeechEncoder, a dual-stream, multi-view foundation architecture coupled with a Parallel Omni-Adapter framework. In the first stage, the encoder models continuous phonetic context and spectral-temporal acoustics in parallel, harmonizing both views through asymmetric stop-gradient cross-modal fusion and curriculum-guided uncertainty weighting to prevent gradient interference across self-supervised and demographic objectives. In the second stage, the pre-trained foundation model is strictly frozen, and lightweight, task-decoupled adapters with adversarial confounder pruning are attached to isolate mutually independent subspaces for speaker identity, language, gender, age, and non-autoregressive speech recognition. Extensive evaluations across in-domain, open-set, and zero-shot cross-domain benchmarks demonstrate that our frozen dual-stream representation outperforms specialized single-task baselines in cross-lingual speaker verification and language identification while eliminating training-volume bias across underrepresented languages. Furthermore, by compressing sequential audio context by an order of magnitude while operating over 35x faster than real-time, the M3LM-SpeechEncoder provides an equitable, computationally agile auditory perceiver primed for direct integration with generative Text LLMs.
	A Systematic Survey of Modality Fusion Strategies in Multimodal Multilingual Multitask Speech-Text Transformers Barathi Ganesh HB , Michal Ptaszynski, Reshma Unnikrishnan, Meenakshi, Jairam R Information Fusion [abs] [pdf] [code] This survey reviews recent advances in Multimodal Multilingual Multitask transformers that jointly process speech and text, with a focus on their role as information fusion systems. We introduce a six‑dimensional taxonomy covering architecture type, pre‑training objectives, modality integration strategies, training data regimes, language coverage and task support. By using it we organised more than forty representative models, ranging from encoder‑only self supervised learning encoders (wav2vec 2.0, XLS‑R) to unified encoder-decoder systems (mSLAM, Mu2SLAM, SeamlessM4T, Whisper) and emerging audio‑aware large language models (SpeechGPT, SALMONN, Qwen‑Audio, SpeechLM). Benchmarks such as LibriSpeech, CoVoST 2, MuST‑C, FLEURS, VoxPopuli, Multilingual LibriSpeech, Common Voice, ML‑SUPERB 2.0, AudioBench, BLASER and HPSU, were used to compare these architectures across speech-text tasks, and quantify where current designs succeed and where they fail, especially on low‑resource and typologically distant languages, code‑switched speech, long‑form audio and paralinguistic cues. We then analyse key limitations like speech-text modality gap, quadratic computational cost, data imbalance, evaluation fragmentation and fairness concerns. It also proposes a research road-map spanning improved cross‑modal alignment, more efficient and modular fusion architectures, stronger low‑resource support and standardised multilingual benchmarks. Overall, the survey provides an architectural and algorithmic perspective on speech-text fusion that outlines concrete steps towards robust, equitable and real‑time multimodal communication systems.
	Overcoming Orthographic Discrepancies via Algorithmic Translinear Pipelines and Phylogenetic Script Mapping Barathi Ganesh HB , Michal Ptaszynski, Meenakshi, Jairam R WMT EMNLP 2026 [abs] [pdf] [code] This paper details the submission of team TIP-RBG-AI for the WMT26 Indic-MT shared task. Addressing severe data sparsity, unstandardized orthographies, and script-level discrepancies across eleven Indic languages, we introduce a robust, resource-agnostic cross-lingual framework. Rather than relying on standard downstream parameter fine-tuning, our methodology optimizes zero-shot inference topologies across three massively multilingual machine translation architectures: NLLB-200, MADLAD-400, and IndicTrans2. To systematically resolve vocabulary mismatch and tokenizer fragmentation, we implement a three-tiered preprocessing pipeline comprising baseline untuned direct inference for native scripts, translinear normalization via algorithmic back-transliteration for Romanized textual representations, and phylogenetic script mapping for undocumented, scriptless vernaculars. Empirical evaluations demonstrate that structurally aligning orthographic decoding environments and exploiting cross-lingual family networks substantially enriches semantic fidelity. Without executing a single downstream parameter update, our purely zero-shot framework secured multiple podium placements, including second-place finishes in the Bodo and Manipuri (Meitei Mayek) to English tracks, validating the immense potential of frontend representation alignment in extreme low-resource translation.
	Disentangled Speech Encoder: A Robust Encoder with Dynamic Adapter for Language Identification Barathi Ganesh HB , Jairam R, Michal Ptaszynski, Reshma U, Jyothish Lal G, Premjith B TidyLang Odyssey 2026 [abs] [pdf] [code] Spoken Language Recognition (SLR) in multilingual open set tasks is challenging due to the overlap between linguistic and speaker specific traits. Fine-tuning large foundation models distorts phonetic representations or leads to overfitting, limiting generalization to unseen speakers. To address this, we propose DisentangLID, a robust adaptation method for the TidyLang Challenge integrating a frozen dual branch backbone with a lightweight adapter and adversarial learning to enforce speaker invariant representations. The method aligns semantic representations from Wav2Vec-2.0-BERT with acoustic features from a ResNet34 encoder while preserving pretrained space. A key contribution is a controlled adaptation strategy that prevents feature drift while enabling task specific refinement. Dynamic under sampling and a two stage adversarial scheme improve robustness. The model achieves 96.01 micro and 86.92 macro accuracy on validation, and 90.33 micro, 83.43 macro accuracy on the test set.
	AURA-ST: Acoustic-Unconstrained Residual Architecture for Speech Translation. Barathi Ganesh HB , Michal Ptaszynski, Reshma U, Jairam R IWSLT 2026 [abs] [pdf] [code] We present AURA-ST, a three-stage modular pipeline for low-resource speech-to-text translation submitted to the IWSLT 2026 African-Celtic Track~1. The architecture bypasses traditional cross-attention between audio and text modalities by treating projected acoustic representations as a native token prefix to a frozen large language model. A dual-stream encoder captures linguistic and paralinguistic features via a jointly trained semantic and a paralinguistic encoder. A convolutional subsampler then bridges the modality gap through a 4x temporal compression and a linear projection into the LLM embedding space. Finally, a MLP-targeted Low-Rank Adaptation adapter fine-tunes the frozen Gemma-4-E2B backbone for translation without catastrophic forgetting of base language model knowledge. We further identify and resolve the incompatibility between standard PEFT attention-level adapter injection and the Gemma-4 Per-Layer Embedding architecture that tends to cause gradient isolation. Trained on the IWSLT 2026 Track 1 data covering Hausa, Igbo, and Yoruba, the final system achieves a best proxy teacher-forced SacreBLEU of 91.29 on the validation set at Phase 3, with Phase~1 speech encoder validation loss converging to 0.651.
	SMM4H-HeaRD 2025: LLMs in Healthcare Applications. Barathi Ganesh HB , Anthony Vijay M., Naren Kishor S., Sharmila B., Jairam R., Jyothish Lal G. ICWSM 2025 [abs] [pdf] [code] This paper presents a lightweight and unified LLM-based system designed for the SMM4H 2025 Task 4: Detection of Insomnia in Clinical Notes. We adopt a single-prompt inference approach using open-source language models constrained to 4B parameters for computational efficiency. On the official test set, RBG-AI achieved an F1-score of 0.9462 (Subtask 1), 0.75 (Subtask 2A), and 0.4631 (Subtask 2B), outperforming baseline metrics. Our results highlight the effectiveness of structured prompting and hybrid rule integration in clinical NLP applications.

Teaching Experience

Coursework

• CN611: Computational Linear Algebra and its Applications
• CN613: Computational Optimization Theory - Linear and Non-Linear Methods
• CN624: Probability and Graphical Models
• CN614: Advanced Data Structures and Algorithms
• SLMA101: Distributed Algorithms and Optimization
• CN733: Neural Networks and Deep Learning
• CN709: Applied Computational Linguistics
• CN736: Text Analytics
• SLCN101: Deep Learning for Natural Language Processing
• CN705: Speech Processing
• CN622: Wavelet Theory and Pattern Classification

Certifications

• “Neural Networks and Deep Learning” acquired from deeplearning.ai through coursera.
• “Deep Learning Using Keras” acquired from udemy.
• “Maxima and Minima concepts : Applications of Derivatives” acquired from udemy.
• “Deep Learning Prerequisites: The Numpy Stack in Python” acquired from udemy.
• “Machine Learning Foundation” acquired from “Tata Consultancy Services” through Tata Consultancy Services internal certification program.
• “Statistics and R” acquired from Harvard through edx.
• “High-Dimensional Data Analysis” acquired from Harvard through edx.
• “Data Mining with Weka” acquired from the University of Waikato.
• “Hadoop Fundamentals” acquired from IBM.
• “Spark Fundamentals” acquired from IBM.
• “Introduction to R” acquired from DataCamp.
• “Introduction to Data Analysis using R” acquired from Big Data University.
• “Introduction to Python” acquired from DataCamp.

International Shared Tasks Participated

• Machine Translation (EMNLP WMT-2026)
• African & Celtic Speech-to-Text Translation track (IWSLT-2026)
• Speaker-Controlled Language Recognition (Odyssey - 2026 TidyLang)
• Audio Encoder Capability Challenge for Large Audio Language Models (InterSpeech-2026 AECC)
• Cross-Lingual Speaker Verification (InterSpeech-2026 TidyVoice)
• Multilingual Automatic Detection of Reclamation of Slurs in the LGBTQ+ Context (MultiPRIDE)
• Fallacy Detection in Italian Social Media Texts(FadeIT)
• Machine Translation (EMNLP WMT-2025)
• Detection of insomnia in clinical notes (SMM4H-HeaRD 2025)
• First Security and Privacy Analytics Anti-Phishing Shared Task (IWSPA-AP 2018)
• Semantic Evaluation Exercises: Affect in Tweets (SemEval-2018)
• PAN @ Conference and Labs of Evaluation Forum: Author Profiling (PAN-2017)
• Forum for Information Retrieval and Evaluation (FIRE-2017): Gender Identification in Russian Texts (RusProfiling)
• Forum for Information Retrieval and Evaluation (FIRE-2017): Information Retrieval from Legal Documents (IRLeD)
• 2nd Social Media Mining for Health Application shared task (AMIA 2017)
• Sentiment Analysis for Indian Languages: Code Mixed (SAIL 2017)
• The 8th International Cybersecurity Data Mining Competition (CDMC-2017)
• Semantic Evaluation Exercises: Semantic Textual Similarity (SemEval-2016)
• PAN @ Conference and Labs of Evaluation Forum: Author Profiling (PAN-2016)
• Forum for Information Retrieval and Evaluation (FIRE-2016): Mixed Script Information Retrieval (MSIR)
• Forum for Information Retrieval and Evaluation (FIRE-2016): Consumer Health Information Search (CHIS)
• Forum for Information Retrieval and Evaluation (FIRE-2016): Code Mixed Entity Extraction – Indian Languages (CMEE – IL)
• Forum for Information Retrieval and Evaluation (FIRE-2014): Named Entity Recognition Indian Languages (NER)
• Named Entity rEcognition and Linking: Named Entity Recognition and Linking (#Micropost2015 NEEL)

Invited Talks

• Topic: “Data Preprocessing and Feature Engineering: The Key to Successful Models” in the Faculty Development Programme at Dr. B.R. Ambedkar Institute of Technology Sri Vijyapuram Andaman & Nicobar Islands. 20, Jan 2025.
• Topic: “Jailbreaking LLM” at Amrita University, Coimbatore. 18, October 2024.
• Topic: “Cyber Security in Industry 4.0” in TEQIP-III Sponsored Workshop on “Deep Learning for Big Data and Cyber Security Applications” at National Institute of Technology, Surathkal, July 2019.
• Topic: “Workshop on Machine Learning” in National Conference on Recent Trends in Computing and Communications NCRTCC’18 at Adi Shankara Institute of Engineering and Technology, February 2018.
• Topic: “Natural Language Processing with Deep Learning” in the Faculty Development Program (FDP) at Vidya Academy of Science & Technology, Thrissur. 20, Jan 2017.
• Topic: “Deep Learning with Python” in the Faculty Development Program (FDP) at Mepco Schlenk Engineering College, Sivakasi. 16-17, Jan 2017.
• Topic: “Dyanamic Mode Decomposition” from scratch . In the workshop on Data -Driven Modelling 2017 at Computational Engineering and Networking (CEN), Amrita School of Engineering, Coimbatore. 8-9, Jan 2017.
• Topic: “Deep Learning for BioInformatics” in the workshop on “DeepChem 2017: Deep Learning & NLP for Computational Chemistry, Biology & Nano-materials”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, December 22-24, 2017.
• Topic: “Dynamic Mode Decomposition in Epidemiology” in the Workshop on “A Refresher experiential course on linear algebra and Optimization for Most Modern Signal processing and pattern classification”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, 25-11,26-11 and 27-11, 2017.
• Topic: “Natural Language Processing for Cyber Security” in the workshop on “AISec 2017 Workshop: Modern Artificial Intelligence (AI) and Natural Language Processing (NLP) Techniques for Cyber Security”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, Saturday, October 28, 2017.
• Topic: “A Journey to Text Analytics and Recent Trends in Text Analytics”, TEQIP sponsored Faculty Development Program on “Research Directions in Data and Text Mining”, organized by Government Engineering College, Idukki on January 2016.
• Topic: “Advanced Analytics in Text and Data Mining”, Tata Consultancy Services sponsored Faculty Development Program on “Text and Data Mining” held at Tata Consultancy Services, Kochi on April 2016.
• Topic: “Computation differs, not Optimization”, TEQIP Sponsored Faculty Development Program on “Contemporary Developments in Optimization Techniques and its Applications”, organized by TKM College of Engineering, Kollam in May 2016.
• Topic: “Web Mining through Data and Text Mining”, Association Inauguration “ORBIT 2K16” conducted by Viswa Jyothi College of Engineering and Technology, Vazhakulam on August 2016.

Workshops and Shared Tasks Conducted

• “Generative AI” workshop for Advanced Data Science Students at St Joseph’s College Of Engineering, Chennai. 01 to 07, April 2025.
• “Generative AI from Essential AI” workshop at Sri Ramakrishna Engineering College, Coimbatore. Febrauary 23 - 24, 2024.
• The “AI Essentials and Fundamentals” workshop at Amrita University, Coimbatore. 20 & 21, April 2022.
• Conducted International Shared Task “Information Extractor for Conversational Systems in Indian Languages (IECSIL)” at Forum for Information Retrieval Evaluation 2018.
• Organized a Workshop on “DeepChem 2017: Deep Learning & NLP for Computational Chemistry, Biology & Nano-materials”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, December 22-24, 2017.
• Organized a Workshop on “Blockchain 2017 Workshop: Blockchain and Machine Learning”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, Saturday December 16, 2017.
• Organized a Workshop on “A Refresher experiential course on linear algebra and Optimization for Most Modern Signal processing and pattern classification”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, 25-11,26-11 and 27-11, 2017.
• Organized a Workshop on “DeepSci 2017 Workshop: Deep Learning for Healthcare and Financial Data Analytics”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, Saturday, November 11, 2017.
• Organized a Workshop on “AISec 2017 Workshop: Modern Artificial Intelligence (AI) and Natural Language Processing (NLP) Techniques for Cyber Security”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University, Saturday, October 28, 2017.
• Advisory Committee member of International Shared Task “Detecting Paraphrases in Indian Languages (DPIL)”, Conducted as a part of Forum for Information Retrieval and Evaluation – 2016 by Amrita Vishwa Vidyapeetham University.
• Advisory Committee member of International Shared Task & Workshop on “Machine Translation system in Indian Languages (MTIL)”, Conducted by Amrita Vishwa Vidyapeetham University.
• Session Chair member of International Shared Task on “Indian Native Language Identification (INLI PAN @ FIRE 2017)”, Conducted by the Department of Computational Engineering and Networking, Amrita Vishwa Vidyapeetham University.

Skills

Last updated on 2026-10-31