Persian NLP

Some Persian NLP projects of mine

Persian LG Syntax Parser - based on the Link Grammar parser

PersianLG code

Perstem - Persian Stemmer and Morphological Analyzer (also does character-set conversion, and some part-of-speech tagging)
Persian Lexicon Project
Persian character set encoding mappings (pdf) - (LaTeX source)
Convert to and from different character sets (old)
Kayhan 2005 word frequency count - top 10,000 words (UTF-8)
Kayhan 2005 word frequency count - top 10,000 words (Romanized - Dehdari)

Here's how many of the Persian programs fit together:

Using an input text like "من کتاب‌های تو را نمی‌بینم" in Unicode HTML decimal form

Orthography

You can convert the text from one character-set encoding to another, including to and from: Romanized, ArabTeX, Windows-1256, ISIRI 3342, UTF-8, and Unicode HTML numeric entities.

For example, from romanized text to UTF-8:

echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl --nostem -i roman -o utf8

and the output:

من کتاب‌های تو را نمی‌بینم

Or from Perso-Arabic script HTML numeric character references to ArabTeX:

echo "من کتاب‌های تو را نمی‌بینم" | perl perstem.pl -d -i unihtml -o arabtex

and the output:

mn ktAb\hspace{0ex}hAI tU rA nmI\hspace{0ex}bInm

This could then be inserted into a LaTeX document as:

  \documentclass{article}
  \usepackage{arabtex}
  
  \begin{document}
  \setfarsi
  \novocalize
  \< mn ktAb\hspace{0ex}hAI tU rA nmI\hspace{0ex}bInm >
  \end{document}

Morphology

You can morphologically analyze this directly:

echo "من کتاب‌های تو را نمی‌بینم" | perl perstem.pl -l -i unihtml

and the output:

mn ktAb_+-hA_+e tu rA n+_mi-+_bin_+m

Or you can analyze this as romanized text:

echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl -l

and the output:

mn ktAb_+-hA_+e tu rA n+_mi-+_bin_+m

You can analyze the text with part-of-speech tags (only for word containing inflexional morphemes):

echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl -l -p

and the output:

mn ktAb_+-hA_+e/N+PL+EZ tu rA n+_mi-+_bin_+m/V+NEG+DUR+1S

You can stem the words (hence the name):

echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl -w

and the output:

mn ktAb tu rA bin

You can remove morpheme links:

echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl

and the output:

mn ktAb hA e tu rA n mi bin m

Syntax

This text can then go into the LG syntax parser:

source persianlg.sh
echo "mn ktAb-hAi tu rA nmi-binm" | perl perstem.pl | persianlg

or the short form:

persianparse.sh "mn ktAb-hAi tu rA nmi-binm"

and the output of either input is:

    +--------------------------------Wi-------------------------------+
    |       +---------------------------Spn1--------------------------+
    |       |      +---------------------On---------------------+     |
    |       |      +-----------PA-----------+                   |     |
    |       |      +-----EZ-----+           |     +----VMneg----+     |
    |       |      +-NMSp-+     +--M-+      |     |      +-VMdur+-VMP-+
    |       |      |      |     |    |      |     |      |      |     |
LEFT-WALL mn.pn ktAb.n hA.nms e.ez tu.pn rA.acc n.vmn mi.vmd bin.vp m.vmp

You can input Perso-Arabic script directly:

source persianlg.sh
echo "من کتاب‌های تو نمی‌بینم" | perl perstem.pl -i unihtml | persianlg

which will result in the same output:

    +--------------------------------Wi-------------------------------+
    |       +---------------------------Spn1--------------------------+
    |       |      +---------------------On---------------------+     |
    |       |      +-----------PA-----------+                   |     |
    |       |      +-----EZ-----+           |     +----VMneg----+     |
    |       |      +-NMSp-+     +--M-+      |     |      +-VMdur+-VMP-+
    |       |      |      |     |    |      |     |      |      |     |
LEFT-WALL mn.pn ktAb.n hA.nms e.ez tu.pn rA.acc n.vmn mi.vmd bin.vp m.vmp

Further information on Perstem

The command "perl perstem.pl --help" gives the following usage information:


Usage:    perl perstem.pl [options] < input > output

Function: Stemmer and morphological analyzer for the Persian language (Farsi).
          Inflexional morphemes are separated from their roots.

Options:
  -d, --nostem           Don't stem -- mostly for character-set conversion
  -h, --help             Print usage
  -i, --input <type>     Input character encoding type {cp1256,isiri3342,utf8,unihtml}
  -l, --links            Show morphological links
  -n, --noroman          Delete all non-Arabic script characters (eg. HTML tags)
  -o, --output <type>    Output character encoding type {arabtex,cp1256,isiri3342,utf8,unihtml}
  -p, --pos              Tag words for parts of speech
  -r, --recall           Increase recall by parsing ambiguous affixes
  -t, --tokenize         Tokenize punctuation
  -u, --unvowel          Remove short vowels
  -v, --version          Print version
  -w, --root             Return only word roots
  -z, --zwnj             Insert Zero Width Non-Joiners where they should be