Sign in
← Back to search
github Not archived

LanguageMachines/ucto

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --

2 awesome lists

Quick read

Stars
72
Forks
14
Open issues
12
Commits
1,591

Activity and growth

Latest capture 2026-11-09 09:48

Stars · last 7 days
0 0.0%
Commits · last 7 days
0 0.0%
Stars since tracking
0
Stored snapshots
1

Compare the starting count and capture dates, not just the percentage. How to read GitHub star growth. Missing history is not zero growth.

Classification

Topics

#computational-linguistics #folia #language #natural-language-processing #nlp #punctuation #tokeniser

Metadata

Language
C++
License
GPL-3.0
Default branch
master
Created
2013-03-26
First commit
2011-03-31
Last pushed
2026-08-20
GitHub updated
2026-08-20
Last synced
2026-11-09 09:48
Stack scanned
2026-11-09 09:48
Archived
No

Archive status reflects the last catalog sync, not a maintenance assessment. Check recent releases, issue responses, and fit for your project with the repository shortlisting checklist.

Sign in to like LanguageMachines/ucto
Compare this repository README Open website View on GitHub

AI development signals

0 paths

Agent instructions and tool configuration found in this repository.

No config files detected.

Growth history

Tracked growth

1 observed capture since 2026-11-09. Observed captures are shown by default.

Stars from first capture 0

Chart data

Observed captures only

Time horizon

All tracked data

Custom date range

Stars history

Observed snapshots

Commits history

Observed snapshots

Similar repositories

Nearest indexed repositories by embedding similarity.

proycon/python-ucto

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

31 stars
Cython 2 awesome lists

LanguageMachines/frog

Frog is an integration of memory-based natural language processing (NLP) modules developed for Dutch. All NLP modules are based on Timbl, the Tilburg memory-based learning software package.

83 stars
C++ 2 awesome lists

proycon/python-frog

Python bindings to the dutch NLP tool Frog (pos tagger, lemmatiser, NER tagger, morphological analysis, shallow parser, dependency parser)

49 stars
Cython 2 awesome lists

ufal/udpipe

UDPipe: Trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files

425 stars
C++ 1 awesome list

LanguageMachines/PICCL

A set of workflows for corpus building through OCR, post-correction and normalisation

51 stars
Python 1 awesome list