instant-segment/README.md

107 lines
3.6 KiB
Markdown
Raw Normal View History

2021-06-05 21:30:22 +00:00
![Cover logo](https://raw.githubusercontent.com/InstantDomain/instant-segment/main/cover.svg)
2020-12-16 09:38:59 +00:00
2021-04-29 09:12:42 +00:00
# Instant Segment: fast English word segmentation in Rust
2020-06-17 20:11:06 +00:00
2020-12-16 09:44:56 +00:00
[![Documentation](https://docs.rs/instant-segment/badge.svg)](https://docs.rs/instant-segment/)
[![Crates.io](https://img.shields.io/crates/v/instant-segment.svg)](https://crates.io/crates/instant-segment)
2021-04-29 09:12:42 +00:00
[![PyPI](https://img.shields.io/pypi/v/instant-segment)](https://pypi.org/project/instant-segment/)
2020-12-15 20:02:22 +00:00
[![Build status](https://github.com/InstantDomainSearch/instant-segment/workflows/CI/badge.svg)](https://github.com/InstantDomainSearch/instant-segment/actions?query=workflow%3ACI)
2020-06-17 20:11:06 +00:00
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE-APACHE)
2021-04-29 09:12:42 +00:00
Instant Segment is a fast Apache-2.0 library for English word segmentation. It
is based on the Python [wordsegment][python] project written by Grant Jenks,
2020-06-17 20:11:06 +00:00
which is in turn based on code from Peter Norvig's chapter [Natural Language
2021-04-29 09:12:42 +00:00
Corpus Data][chapter] from the book [Beautiful Data][book] (Segaran and
Hammerbacher, 2009).
2020-06-17 20:11:06 +00:00
For the microbenchmark included in this repository, Instant Segment is ~100x
2021-08-18 20:18:56 +00:00
faster than the Python implementation. The API was carefully constructed
so that multiple segmentations can share the underlying state to allow parallel
usage.
2021-04-29 09:12:42 +00:00
## How it works
Instant Segment works by segmenting a string into words by selecting the splits
with the highest probability given a corpus of words and their occurrences.
For instance, provided that `choose` and `spain` occur more frequently than
`chooses` and `pain`, and that the pair `choose spain` occurs more frequently
than `chooses pain`, Instant Segment can help identify the domain
`choosespain.com` as `ChooseSpain.com` which more likely matches user intent.
2021-08-18 20:23:38 +00:00
Read about [how we built and improved][story] Instant Segment for use in production
at [Instant Domain Search](https://instantdomainsearch.com/) to help our users
find relevant domains they can register.
2021-04-29 09:12:42 +00:00
## Using the library
### Python **(>= 3.9)**
```sh
pip install instant-segment
```
### Rust
```toml
[dependencies]
instant-segment = "0.8.1"
```
### Examples
The following examples expect `unigrams` and `bigrams` to exist. See the
examples ([Rust](./instant-segment/examples/contrived.rs),
[Python](./instant-segment-py/examples/contrived.py)) to see how to construct
these objects.
```python
import instant_segment
segmenter = instant_segment.Segmenter(unigrams, bigrams)
search = instant_segment.Search()
segmenter.segment("instantdomainsearch", search)
print([word for word in search])
--> ['instant', 'domain', 'search']
```
```rust
use instant_segment::{Search, Segmenter};
use std::collections::HashMap;
let segmenter = Segmenter::from_maps(unigrams, bigrams);
let mut search = Search::default();
let words = segmenter
.segment("instantdomainsearch", &mut search)
.unwrap();
println!("{:?}", words.collect::<Vec<&str>>())
--> ["instant", "domain", "search"]
```
Check out the tests for more thorough examples:
[Rust](./instant-segment/src/test_cases.rs),
[Python](./instant-segment-py/test/test.py)
## Testing
To run the tests run the following:
```
cargo t -p instant-segment --all-features
```
You can also test the Python bindings with:
```
make test-python
```
2020-06-17 20:11:06 +00:00
[python]: https://github.com/grantjenks/python-wordsegment
[chapter]: http://norvig.com/ngrams/
2021-08-18 20:23:38 +00:00
[story]: https://instantdomainsearch.com/engineering/instant-word-segmentation-with-rust
2020-06-17 20:11:06 +00:00
[book]: http://oreilly.com/catalog/9780596157111/
[distributed]: https://catalog.ldc.upenn.edu/LDC2006T13
[issues]: https://github.com/InstantDomainSearch/instant-segment/issues