Awesome Nlp

A Language Processing Toolkit for Persian

The tool to make NLP datasets ready to use

BLLIP reranking parser (also known as Charniak-Johnson parser, Charniak parser, Brown reranking parser) See http://pypi.python.org/pypi/bllipparser/ for Python module.

FastFit ⚡ When LLMs are Unfit Use FastFit ⚡ Fast and Effective Text Classification with Many Classes

Repository for research in the field of Responsible NLP at Meta.

DaNLP is a repository for Natural Language Processing resources for the Danish Language.

Morfessor is a tool for unsupervised and semi-supervised morphological segmentation

Tools, wrappers, etc... for data science with a concentration on text processing

StereoSet: Measuring stereotypical bias in pretrained language models

Comprehensive NLP Evaluation System

초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

Sentiment Classification using Word Sense Disambiguation

Thai Word-Segmentation with LSTM in Tensorflow

Japanese Natural Langauge Processing Libraries

This repository contains the data and code introduced in the paper "CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models" (EMNLP 2020).

[EMNLP-2024] ⚓️ Sailor: Open Language Models for South-East Asia

Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.

OpenIE6 system

Pretraining, fine-tuning and evaluation scripts for IndicBERT-v2 and IndicXTREME

Pretraining transformer based Thai language models

IndoLEM is a comprehensive Indonesian NLU benchmark, comprising three pillars NLP task: morpho-syntax, semantic, and discourse. Presented in COLING 2020.

FoLiA Linguistic Annotation Tool -- Flat is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.github.io/folia), a rich XML-based format for linguistic annotation. Flat allows users to view annotated FoLiA documents and enrich these documents with new annotations, a wide variety of linguistic annotation types is supported through the FoLiA paradigm.

A collaborative project to collect datasets in SEA languages, SEA regions, or SEA cultures.

An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!

A Multilingual Latent Dirichlet Allocation (LDA) Pipeline with Stop Words Removal, n-gram features, and Inverse Stemming, in Python.

A keyphrase extractor for Persian

A Domain Specific Language (DSL) for building language patterns. These can be later compiled into spaCy patterns, pure regex, or any other format

No description.

InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.

Code and models for "The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models". EACL 2021, WANLP.

PyTorch implementation of the end-to-end coreference resolution model with different higher-order inference methods.

Spanish Billion Word Corpus and Embeddings

Python bindings to the dutch NLP tool Frog (pos tagger, lemmatiser, NER tagger, morphological analysis, shallow parser, dependency parser)

spanlp: nlp applied for spanish vulgarity. A fast, robust Python library to check for profanity or offensive language in Spanish strings. It contains all the rude words of Spanish-speaking countries.

Thai Word Segmentation and Part-of-Speech Tagging with Deep Learning

A character-wise tokenizer for morphologically rich languages

Alpino parser and related tools for Dutch

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

DRF-based API server for Shoonya platform

Natural language toolkit for Indonesian Language (Bahasa)

An Indonesian word embedding demo

Tiny QA Benchmark++ a micro-benchmark suite (52-item gold + on-demand multilingual synthetic packs), generator CLI, and CI-ready eval harness for ultra-fast LLM smoke-testing & regression-catching.

An extensive Python library for dealing with FoLiA (Format for Linguistic Annotation) documents, a rich XML-based format for linguistic annotation finding application in Natural Language Processing (NLP). This library was formerly part of PyNLPl.

Aspect based sentiment analysis for Hindi

Bilingual RAG evaluation benchmark for culturally grounded English/Uzbek retrieval