Awesome Nlp

Beautiful visualizations of how language differs among document types.

Basic Utilities for PyTorch Natural Language Processing (NLP)

Code and data accompanying Natural Language Processing with PyTorch published by O'Reilly Media https://amzn.to/3JUgR2L

BERT score for text generation

Underthesea - AI Assistant

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

This repository includes the official implementation of OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs.

Data and tools for generating and inspecting OLMo pre-training data.

Fast Neural Machine Translation in C++

Pre-trained ELMo Representations for Many Languages

Korean BERT pre-trained cased (KoBERT)

LongBench v2 and LongBench (ACL 25'&24')

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

Morphological analyzer / inflection engine for Russian and Ukrainian languages.

Expanding natural instructions

TruthfulQA: Measuring How Models Imitate Human Falsehoods

Portuguese pre-trained BERT models

Natural Language Toolkit for Indic Languages aims to provide out of the box support for various NLP tasks that an application developer might need

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

Facebook Low Resource (FLoRes) MT Benchmark

Pre-trained Transformers for Arabic Language Understanding and Generation (Arabic BERT, Arabic GPT2, Arabic ELECTRA)

Fast topic modeling platform

The first-ever vast natural language processing benchmark for Indonesian Language. We provide multiple downstream tasks, pre-trained IndoBERT models, and a starter code! (AACL-IJCNLP 2020)

Dataset and codes for ACL 2019 DocRED: A Large-Scale Document-Level Relation Extraction Dataset.

Resources and tools for Indian language Natural Language Processing

Hierarchical unsupervised and semi-supervised topic models for sparse count data with CorEx

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

Accurately generate all possible forms of an English word e.g "election" --> "elect", "electoral", "electorate" etc.

NLP in Python with Deep Learning

A suite of Arabic natural language processing tools developed by the CAMeL Lab at New York University Abu Dhabi.

REBEL is a seq2seq model that simplifies Relation Extraction (EMNLP 2021).

The hands-on NLTK tutorial for NLP in Python

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

Deliver safe & effective language models

Library for translating between 200 languages. Built on 🤗 transformers.

Translation models for 22 scheduled languages of India

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

KSS: Korean String processing Suite

A package to evaluate factuality of long-form generation. Original implementation of our EMNLP 2023 paper "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation"

potato: the portable annotation tool

No description.

🤗 ParsBERT: Transformer-based Model for Persian Language Understanding

South-East Asia Large Language Models

UDPipe: Trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files

Indonesian stemmer. Python port of PHP Sastrawi project.

A Python library for Interpretable Machine Learning in Text Classification using the SS3 model, with easy-to-use visualization tools for Explainable AI :octocat:

Snips NLU rust implementation

An NLP library for the Urdu language. It comes with a lot of battery included features to help you process Urdu data in the easiest way possible.

A collaborative project to collect datasets in Indonesian languages.

Python Vietnamese Core NLP Toolkit