Awesome Nlp

LLM inference in C/C++

Robust Speech Recognition via Large-Scale Weak Supervision

A high-throughput and memory-efficient inference and serving engine for LLMs

SGLang is a high-performance serving framework for large language models and multimodal models.

💫 Industrial-strength Natural Language Processing (NLP) in Python

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Label Studio is a multi-type data labeling and annotation tool with standardized output format

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Train transformer language models with reinforcement learning.

State-of-the-Art Embeddings, Retrieval, and Reranking

Supercharge Your LLM Application Evaluations 🚀

A framework for few-shot evaluation of language models.

A very simple framework for state-of-the-art Natural Language Processing (NLP)

Unsupervised text tokenizer for Neural Network-based text generation.

An open-source NLP research library, built on PyTorch.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Large Language Model Text Generation Inference

YSDA course in Natural Language Processing

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Language Technology Platform

Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

Vietnamese TTS with instant voice cloning • On-device • Real-time CPU inference • 48kHz audio quality • Chuyển văn bản thành giọng nói tiếng Việt • Text to speech tiếng Việt • TTS tiếng Việt

NLP made easy

Unified Schema-Based Information Extraction

Doing simple retrieval from LLM models at various context lengths to measure accuracy

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

NLP, before and after spaCy

Basic Utilities for PyTorch Natural Language Processing (NLP)

BERT score for text generation

Underthesea - AI Assistant

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

This repository includes the official implementation of OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs.

Data and tools for generating and inspecting OLMo pre-training data.

Persian NLP Toolkit

Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

The Classical Language Toolkit

Deliver safe & effective language models

Library for translating between 200 languages. Built on 🤗 transformers.

potato: the portable annotation tool

A Python library for Interpretable Machine Learning in Text Classification using the SS3 model, with easy-to-use visualization tools for Explainable AI :octocat:

FastFit ⚡ When LLMs are Unfit Use FastFit ⚡ Fast and Effective Text Classification with Many Classes

OpenIE6 system

A Multilingual Latent Dirichlet Allocation (LDA) Pipeline with Stop Words Removal, n-gram features, and Inverse Stemming, in Python.

A keyphrase extractor for Persian

A Domain Specific Language (DSL) for building language patterns. These can be later compiled into spaCy patterns, pure regex, or any other format

spanlp: nlp applied for spanish vulgarity. A fast, robust Python library to check for profanity or offensive language in Spanish strings. It contains all the rude words of Spanish-speaking countries.