Awesome Nlp

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Robust Speech Recognition via Large-Scale Weak Supervision

A high-throughput and memory-efficient inference and serving engine for LLMs

🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production

SGLang is a high-performance serving framework for large language models and multimodal models.

💫 Industrial-strength Natural Language Processing (NLP) in Python

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Label Studio is a multi-type data labeling and annotation tool with standardized output format

Library for fast text representation and classification.

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

Train transformer language models with reinforcement learning.

A very simple framework for state-of-the-art Natural Language Processing (NLP)

Unsupervised text tokenizer for Neural Network-based text generation.

Foundational Models for State-of-the-Art Speech and Text Translation

An open-source NLP research library, built on PyTorch.

Large Language Model Text Generation Inference

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

Language Technology Platform

Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets

TextAttack 🐙 is a Python framework for adversarial attacks, data augmentation, and model training in NLP https://textattack.readthedocs.io/en/master/

Unified Schema-Based Information Extraction

Doing simple retrieval from LLM models at various context lengths to measure accuracy

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

NLP, before and after spaCy

Underthesea - AI Assistant

This repository includes the official implementation of OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs.

Data and tools for generating and inspecting OLMo pre-training data.

Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons

Thai natural language processing in Python

A suite of Arabic natural language processing tools developed by the CAMeL Lab at New York University Abu Dhabi.

potato: the portable annotation tool

Comprehensive NLP Evaluation System

Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.

Python bindings to the dutch NLP tool Frog (pos tagger, lemmatiser, NER tagger, morphological analysis, shallow parser, dependency parser)

A high-accuracy, multilingual rule-based Sentence Boundary Detector (SBD) with a drop-in adapter for pysbd, delivering faster and more accurate sentence segmentation.

Platform for Evaluating and Reviewing of Multilingual Tasks

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

Tiny QA Benchmark++ a micro-benchmark suite (52-item gold + on-demand multilingual synthetic packs), generator CLI, and CI-ready eval harness for ultra-fast LLM smoke-testing & regression-catching.

sentiment analysis for spacy pipeline in python

Bilingual RAG evaluation benchmark for culturally grounded English/Uzbek retrieval