Fast English word segmentation in Rust
Go to file
Dirkjan Ochtman 947e003a48 Store splits instead of string slices 2020-11-25 17:29:13 +01:00
.github/workflows Abstract over test data format code and API 2020-11-25 17:29:13 +01:00
benches Abstract over test data format code and API 2020-11-25 17:29:13 +01:00
data Initial version 2020-05-26 20:07:00 +02:00
src Store splits instead of string slices 2020-11-25 17:29:13 +01:00
tests Abstract over test data format code and API 2020-11-25 17:29:13 +01:00
.gitignore Initial version 2020-05-26 20:07:00 +02:00
Cargo.toml Abstract over test data format code and API 2020-11-25 17:29:13 +01:00
README.md Create initial README (fixes #1) 2020-06-19 13:14:27 +02:00
deny.toml Initial version 2020-05-26 20:07:00 +02:00

README.md

word-segmenters: fast English word segmentation in Rust

Build status License: Apache 2.0

word-segmenters is a fast Apache-2.0 library for English word segmentation. It is based on the Python wordsegment project written by Grant Jenkins, which is in turn based on code from Peter Norvig's chapter Natural Language Corpus Data from the book Beautiful Data (Segaran and Hammerbacher, 2009).

The data files in this repository are derived from the Google Web Trillion Word Corpus, as described by Thorsten Brants and Alex Franz, and distributed by the Linguistic Data Consortium. Note that this data "may only be used for linguistic education and research", so for any other usage you should acquire a different data set.

For the microbenchmark included in this repository, word-segmenters is ~5x faster than the Python implementation. Further optimizations are planned -- see the issues. The API has been carefully constructed so that multiple segmentations can share the underlying state (mainly the unigram and bigram maps) to allow parallel usage.