From 99ddbf736610b359bce5d8bc2cc0a14618b69bc6 Mon Sep 17 00:00:00 2001 From: Dirkjan Ochtman Date: Wed, 2 Jun 2021 14:34:35 +0200 Subject: [PATCH] Update data README --- data/LICENSE-SCOWL | 248 +++++++++++++++++++++++++++++++++++++++++++++ data/README.md | 43 ++++++-- 2 files changed, 285 insertions(+), 6 deletions(-) create mode 100644 data/LICENSE-SCOWL diff --git a/data/LICENSE-SCOWL b/data/LICENSE-SCOWL new file mode 100644 index 0000000..e49dc16 --- /dev/null +++ b/data/LICENSE-SCOWL @@ -0,0 +1,248 @@ +The collective work is Copyright 2000-2018 by Kevin Atkinson as well +as any of the copyrights mentioned below: + + Copyright 2000-2018 by Kevin Atkinson + + Permission to use, copy, modify, distribute and sell these word + lists, the associated scripts, the output created from the scripts, + and its documentation for any purpose is hereby granted without fee, + provided that the above copyright notice appears in all copies and + that both that copyright notice and this permission notice appear in + supporting documentation. Kevin Atkinson makes no representations + about the suitability of this array for any purpose. It is provided + "as is" without express or implied warranty. + +Alan Beale also deserves special credit as he has, +in addition to providing the 12Dicts package and being a major +contributor to the ENABLE word list, given me an incredible amount of +feedback and created a number of special lists (those found in the +Supplement) in order to help improve the overall quality of SCOWL. + +The 10 level includes the 1000 most common English words (according to +the Moby (TM) Words II [MWords] package), a subset of the 1000 most +common words on the Internet (again, according to Moby Words II), and +frequently class 16 from Brian Kelk's "UK English Wordlist +with Frequency Classification". + +The MWords package was explicitly placed in the public domain: + + The Moby lexicon project is complete and has + been place into the public domain. Use, sell, + rework, excerpt and use in any way on any platform. + + Placing this material on internal or public servers is + also encouraged. The compiler is not aware of any + export restrictions so freely distribute world-wide. + + You can verify the public domain status by contacting + + Grady Ward + 3449 Martha Ct. + Arcata, CA 95521-4884 + + grady@netcom.com + grady@northcoast.com + +The "UK English Wordlist With Frequency Classification" is also in the +Public Domain: + + Date: Sat, 08 Jul 2000 20:27:21 +0100 + From: Brian Kelk + + > I was wondering what the copyright status of your "UK English + > Wordlist With Frequency Classification" word list as it seems to + > be lacking any copyright notice. + + There were many many sources in total, but any text marked + "copyright" was avoided. Locally-written documentation was one + source. An earlier version of the list resided in a filespace called + PUBLIC on the University mainframe, because it was considered public + domain. + + Date: Tue, 11 Jul 2000 19:31:34 +0100 + + > So are you saying your word list is also in the public domain? + + That is the intention. + +The 20 level includes frequency classes 7-15 from Brian's word list. + +The 35 level includes frequency classes 2-6 and words appearing in at +least 11 of 12 dictionaries as indicated in the 12Dicts package. All +words from the 12Dicts package have had likely inflections added via +my inflection database. + +The 12Dicts package and Supplement is in the Public Domain. + +The WordNet database, which was used in the creation of the +Inflections database, is under the following copyright: + + This software and database is being provided to you, the LICENSEE, + by Princeton University under the following license. By obtaining, + using and/or copying this software and database, you agree that you + have read, understood, and will comply with these terms and + conditions.: + + Permission to use, copy, modify and distribute this software and + database and its documentation for any purpose and without fee or + royalty is hereby granted, provided that you agree to comply with + the following copyright notice and statements, including the + disclaimer, and that the same appear on ALL copies of the software, + database and documentation, including modifications that you make + for internal use or for distribution. + + WordNet 1.6 Copyright 1997 by Princeton University. All rights + reserved. + + THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON + UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR + IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON + UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT- + ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE + LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY + THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS. + + The name of Princeton University or Princeton may not be used in + advertising or publicity pertaining to distribution of the software + and/or database. Title to copyright in this software, database and + any associated documentation shall at all times remain with + Princeton University and LICENSEE agrees to preserve same. + +The 40 level includes words from Alan's 3esl list found in version 4.0 +of his 12dicts package. Like his other stuff the 3esl list is also in the +public domain. + +The 50 level includes Brian's frequency class 1, words appearing +in at least 5 of 12 of the dictionaries as indicated in the 12Dicts +package, and uppercase words in at least 4 of the previous 12 +dictionaries. A decent number of proper names is also included: The +top 1000 male, female, and Last names from the 1990 Census report; a +list of names sent to me by Alan Beale; and a few names that I added +myself. Finally a small list of abbreviations not commonly found in +other word lists is included. + +The name files form the Census report is a government document which I +don't think can be copyrighted. + +The file special-jargon.50 uses common.lst and word.lst from the +"Unofficial Jargon File Word Lists" which is derived from "The Jargon +File". All of which is in the Public Domain. This file also contain +a few extra UNIX terms which are found in the file "unix-terms" in the +special/ directory. + +The 55 level includes words from Alan's 2of4brif list found in version +4.0 of his 12dicts package. Like his other stuff the 2of4brif is also +in the public domain. + +The 60 level includes all words appearing in at least 2 of the 12 +dictionaries as indicated by the 12Dicts package. + +The 70 level includes Brian's frequency class 0 and the 74,550 common +dictionary words from the MWords package. The common dictionary words, +like those from the 12Dicts package, have had all likely inflections +added. The 70 level also included the 5desk list from version 4.0 of +the 12Dics package which is in the public domain. + +The 80 level includes the ENABLE word list, all the lists in the +ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics +Dictionary" (UKACD), the list of signature words from the YAWL package, +and the 10,196 places list from the MWords package. + +The ENABLE package, mainted by M\Cooper , +is in the Public Domain: + + The ENABLE master word list, WORD.LST, is herewith formally released + into the Public Domain. Anyone is free to use it or distribute it in + any manner they see fit. No fee or registration is required for its + use nor are "contributions" solicited (if you feel you absolutely + must contribute something for your own peace of mind, the authors of + the ENABLE list ask that you make a donation on their behalf to your + favorite charity). This word list is our gift to the Scrabble + community, as an alternate to "official" word lists. Game designers + may feel free to incorporate the WORD.LST into their games. Please + mention the source and credit us as originators of the list. Note + that if you, as a game designer, use the WORD.LST in your product, + you may still copyright and protect your product, but you may *not* + legally copyright or in any way restrict redistribution of the + WORD.LST portion of your product. This *may* under law restrict your + rights to restrict your users' rights, but that is only fair. + +UKACD, by J Ross Beresford , is under the +following copyright: + + Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved. + + The following restriction is placed on the use of this publication: + if The UK Advanced Cryptics Dictionary is used in a software package + or redistributed in any form, the copyright notice must be + prominently displayed and the text of this document must be included + verbatim. + + There are no other restrictions: I would like to see the list + distributed as widely as possible. + +The 95 level includes the 354,984 single words, 256,772 compound +words, 4,946 female names and the 3,897 male names, and 21,986 names +from the MWords package, ABLE.LST from the ENABLE Supplement, and some +additional words found in my part-of-speech database that were not +found anywhere else. + +Accent information was taken from UKACD. + +The VarCon package was used to create the American, British, Canadian, +and Australian word list. It is under the following copyright: + + Copyright 2000-2016 by Kevin Atkinson + + Permission to use, copy, modify, distribute and sell this array, the + associated software, and its documentation for any purpose is hereby + granted without fee, provided that the above copyright notice appears + in all copies and that both that copyright notice and this permission + notice appear in supporting documentation. Kevin Atkinson makes no + representations about the suitability of this array for any + purpose. It is provided "as is" without express or implied warranty. + + Copyright 2016 by Benjamin Titze + + Permission to use, copy, modify, distribute and sell this array, the + associated software, and its documentation for any purpose is hereby + granted without fee, provided that the above copyright notice appears + in all copies and that both that copyright notice and this permission + notice appear in supporting documentation. Benjamin Titze makes no + representations about the suitability of this array for any + purpose. It is provided "as is" without express or implied warranty. + + Since the original words lists come from the Ispell distribution: + + Copyright 1993, Geoff Kuenning, Granada Hills, CA + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + 3. All modifications to the source code must be clearly marked as + such. Binary redistributions based on modified source code + must be clearly marked as modified versions in the documentation + and/or other materials provided with the distribution. + (clause 4 removed with permission from Geoff Kuenning) + 5. The name of Geoff Kuenning may not be used to endorse or promote + products derived from this software without specific prior + written permission. + + THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + SUCH DAMAGE. diff --git a/data/README.md b/data/README.md index 25ea2a3..2d2e964 100644 --- a/data/README.md +++ b/data/README.md @@ -1,7 +1,38 @@ -The data files in this directory are derived from the [Google Web Trillion Word -Corpus][corpus], as described by Thorsten Brants and Alex Franz, and [distributed][distributed] by the -Linguistic Data Consortium. Note that this data **"may only be used for linguistic -education and research"**, so for any other usage you should acquire a different data set. +# Example data files -[corpus]: http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html -[distributed]: https://catalog.ldc.upenn.edu/LDC2006T13 \ No newline at end of file +The data files in this directory are derived from the Google Books Ngram data +([version 3, 2020-02-17][ngrams]) and the [SCOWL][scowl] word list. + +## Recreating these files + +The code used to build the provided `en-unigrams.txt` and `en-bigrams.txt` +is provided as part of this repository. To recreate the files, first +download the input data. Note that this will download about 300 GB of +data and unpack it to about 1.4 TB of data. + +``` +instant-segment $ cd data +data $ python3 grab.py +``` + +After the data has been downloaded, run the `merge` tool to create the word lists: + +``` +instant-segment $ cargo run --release --example merge +``` + +## License + +The SCOWL word list is licensed under a number of licenses detailed in +[LICENSE-SCOWL](./LICENSE-SCOWL), which the website describes as MIT-like. + +The Ngram data provided by Google is available under the Creative Commons +Attribution 3.0 Unported license: + +> This work is licensed under the Creative Commons Attribution 3.0 Unported +> License. To view a copy of this license, visit +> http://creativecommons.org/licenses/by/3.0/ or send a letter to Creative +> Commons, PO Box 1866, Mountain View, CA 94042, USA. + +[ngrams]: https://storage.googleapis.com/books/ngrams/books/datasetsv3.html +[scowl]: http://wordlist.aspell.net/