instant-labs
/
instant-distance
mirror of https://github.com/instant-labs/instant-distance.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Don't index English words

This commit is contained in:
Nicholas Rempel 2021-05-31 10:09:22 -07:00 committed by Dirkjan Ochtman
parent fabe10271d
commit 67778600ef
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
instant-distance-py/examples/translations/translate.py
View File

@ -68,6 +68,11 @@ async def download_build_index():
if lang == "en": if lang == "en":
word_map[value] = embedding word_map[value] = embedding
else: else:
# Don't index words that exist in english
# to improve the quality of the results.
if value in word_map:
continue
# We track values here to build the instant-distance index # We track values here to build the instant-distance index
# Every value is prepended with 2 character language code. # Every value is prepended with 2 character language code.
# This allows us to determine language output later. # This allows us to determine language output later.