747dd22098
Add some minimal documentation
2022-02-28 11:38:50 +01:00
2ef7eb82ba
Bump instant-segment version to 0.9
2022-02-28 11:38:34 +01:00
eac10fb553
Apply clippy suggestion
2022-02-28 11:35:38 +01:00
3709223fa9
Update pyo3 to 0.16
2022-02-28 11:34:57 +01:00
77203fc78c
Update smartstring to 1
2022-02-28 11:34:32 +01:00
c911457226
Leverage simplified Python protocols
2021-11-10 09:41:31 +01:00
2d97878ed7
Update pyo3 requirement from 0.14.1 to 0.15.0
...
Updates the requirements on [pyo3](https://github.com/pyo3/pyo3 ) to permit the latest version.
- [Release notes](https://github.com/pyo3/pyo3/releases )
- [Changelog](https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md )
- [Commits](https://github.com/pyo3/pyo3/compare/v0.14.1...v0.15.0 )
---
updated-dependencies:
- dependency-name: pyo3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-10 09:40:14 +01:00
30a80c4a79
Add minimal pyproject.toml
2021-09-15 16:37:41 +02:00
ef38bedaa2
Improve selection of Python versions
2021-09-01 16:11:13 +02:00
950faa6854
Tweak publish workflow
2021-08-31 16:42:39 +02:00
4c317051ef
Publish platform wheels after tag push
2021-08-31 16:29:17 +02:00
fb2a46ca27
Add Python test job
2021-08-31 16:29:17 +02:00
bd12ef8b69
Fix up python test code
2021-08-31 16:29:17 +02:00
906f202611
Fix some more clippy suggestions
2021-08-31 14:19:36 +02:00
e12390ea7d
py: bump version number to 0.1.4
2021-08-31 14:16:33 +02:00
2babad4a58
Bump version to 0.8.3
2021-08-31 14:15:32 +02:00
f32b42537a
Update links to point to new GitHub org
2021-08-31 14:10:13 +02:00
edfad13ddc
Fix clippy lint
2021-08-31 14:09:23 +02:00
f16306499c
Update README.md
2021-08-18 13:23:38 -07:00
8230ac6ed5
Update README.md
2021-08-18 13:18:56 -07:00
fdd743478e
Update pyo3 requirement from 0.13.2 to 0.14.1
...
Updates the requirements on [pyo3](https://github.com/pyo3/pyo3 ) to permit the latest version.
- [Release notes](https://github.com/pyo3/pyo3/releases )
- [Changelog](https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md )
- [Commits](https://github.com/pyo3/pyo3/compare/v0.13.2...v0.14.1 )
---
updated-dependencies:
- dependency-name: pyo3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-07-05 15:09:48 +02:00
e2f6f5c4a5
Update README.md
2021-06-05 14:30:22 -07:00
bc59c6cf6f
Refactor to make test segmenter more accessible
2021-06-03 16:09:24 +02:00
65b85d9806
Remove old data files
2021-06-03 16:09:24 +02:00
99ddbf7366
Update data README
2021-06-03 16:09:24 +02:00
3c52201fa0
Update test cases to deal with new data
2021-06-03 16:09:24 +02:00
fee2adb995
Add new data files
2021-06-03 16:09:24 +02:00
fcf24c7543
Add Rust code to process ngram data
2021-06-03 16:09:24 +02:00
cc95d39063
Add script to download word list input data
2021-06-03 16:09:24 +02:00
57221b1dd5
Improve test framework to show all failures
2021-06-03 16:09:24 +02:00
e4e773c896
py: bump version to 0.1.3
2021-05-28 15:21:30 +02:00
89c232e3af
py: update crate metadata
2021-05-28 15:20:28 +02:00
7214ffc126
Remove note about planned further optimizations
2021-05-28 14:44:44 +02:00
f081d4b171
py: bump version to 0.1.1
2021-05-28 14:34:04 +02:00
9edd1bc8b7
Bump version number to 0.8.2
2021-05-28 14:31:59 +02:00
85f4f94b53
Use more efficient segmentation strategy
...
Based on the triangular matrix approach as explained here:
https://towardsdatascience.com/fast-word-segmentation-for-noisy-text-2c2c41f9e8da
Use iteration rather than recursion to segment the input forwards
rather than backwards and use a `Vec`-based memoization strategy
instead of relying on a `HashMap` of words. This version is about
4.8x faster, 100 lines of code less and should use much less memory.
2021-05-28 14:30:27 +02:00
541644a329
Upgrade to GitHub-native Dependabot
2021-04-30 09:51:09 +02:00
0ebae2923c
Add license files ( fixes #15 )
2021-04-29 15:34:02 +02:00
9bbb633f1d
Flesh out README ( #14 )
2021-04-29 11:12:42 +02:00
eca12c572f
Bump version number to 0.8.1
2021-04-22 15:08:23 +02:00
bba1de7543
Simplify loop
2021-04-22 15:07:54 +02:00
c21b66ab83
Rename sentence_score() to score_sentence()
2021-04-22 15:04:48 +02:00
62f5b79d6d
py: add Segmenter::sentence_score() method
2021-04-22 15:04:06 +02:00
85035a9b34
Add Segmenter::sentence_score() method
2021-04-22 14:58:06 +02:00
bd014dcc5c
Move logarithm conversion into score()
2021-04-22 14:54:54 +02:00
85038d1f6f
Add files via upload
2021-04-20 11:06:47 -07:00
507e8da5ef
Bump version to 0.8.0
2021-04-01 11:04:42 +02:00
754b0d5692
Revert version number for testing
2021-04-01 10:07:22 +02:00
2d942bbfc9
Box up the BitVec array
...
The `Search::best` field will take about 8000 bytes. In some of our usage
with rayon, this appeared to cause stack overflows. Boxing it up makes the
code slower by about 1-2%, but should hopefully avoid stack overflows.
2021-04-01 10:03:48 +02:00
a4fe0e4039
Bump version to 0.7.2
...
Now that the core crate is in a directory, we no longer needlessly publish
data files on crates.io.
2021-03-24 13:16:49 +01:00
Dirkjan Ochtman
dependabot[bot]
Beau Hartshorne
dependabot-preview[bot]
Nick Rempel