Fast English word segmentation in Rust

natural-language-processing rust segmentation

Go to file

Dirkjan Ochtman 947e003a48 Store splits instead of string slices		2020-11-25 17:29:13 +01:00
.github/workflows	Abstract over test data format code and API	2020-11-25 17:29:13 +01:00
benches	Abstract over test data format code and API	2020-11-25 17:29:13 +01:00
data	Initial version	2020-05-26 20:07:00 +02:00
src	Store splits instead of string slices	2020-11-25 17:29:13 +01:00
tests	Abstract over test data format code and API	2020-11-25 17:29:13 +01:00
.gitignore	Initial version	2020-05-26 20:07:00 +02:00
Cargo.toml	Abstract over test data format code and API	2020-11-25 17:29:13 +01:00
README.md	Create initial README (fixes #1 )	2020-06-19 13:14:27 +02:00
deny.toml	Initial version	2020-05-26 20:07:00 +02:00

README.md

word-segmenters: fast English word segmentation in Rust

word-segmenters is a fast Apache-2.0 library for English word segmentation. It is based on the Python wordsegment project written by Grant Jenkins, which is in turn based on code from Peter Norvig's chapter Natural Language Corpus Data from the book Beautiful Data (Segaran and Hammerbacher, 2009).

The data files in this repository are derived from the Google Web Trillion Word Corpus, as described by Thorsten Brants and Alex Franz, and distributed by the Linguistic Data Consortium. Note that this data "may only be used for linguistic education and research", so for any other usage you should acquire a different data set.

For the microbenchmark included in this repository, word-segmenters is ~5x faster than the Python implementation. Further optimizations are planned -- see the issues. The API has been carefully constructed so that multiple segmentations can share the underlying state (mainly the unigram and bigram maps) to allow parallel usage.