History

Dirkjan Ochtman 65b85d9806 Remove old data files		2021-06-03 16:09:24 +02:00
..
LICENSE-SCOWL	Update data README	2021-06-03 16:09:24 +02:00
README.md	Update data README	2021-06-03 16:09:24 +02:00
en-bigrams.txt	Add new data files	2021-06-03 16:09:24 +02:00
en-unigrams.txt	Add new data files	2021-06-03 16:09:24 +02:00
grab.py	Add script to download word list input data	2021-06-03 16:09:24 +02:00
words.txt	Initial version	2020-05-26 20:07:00 +02:00

README.md

Example data files

The data files in this directory are derived from the Google Books Ngram data (version 3, 2020-02-17) and the SCOWL word list.

Recreating these files

The code used to build the provided en-unigrams.txt and en-bigrams.txt is provided as part of this repository. To recreate the files, first download the input data. Note that this will download about 300 GB of data and unpack it to about 1.4 TB of data.

instant-segment $ cd data
data $ python3 grab.py

After the data has been downloaded, run the merge tool to create the word lists:

instant-segment $ cargo run --release --example merge

License

The SCOWL word list is licensed under a number of licenses detailed in LICENSE-SCOWL, which the website describes as MIT-like.

The Ngram data provided by Google is available under the Creative Commons Attribution 3.0 Unported license:

This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.