HanLP: Han Language Processing
=====
[](https://maven-badges.herokuapp.com/maven-central/com.hankcs/hanlp/)
[](https://github.com/hankcs/hanlp/releases)
[](https://www.apache.org/licenses/LICENSE-2.0.html)
[](https://hub.docker.com/r/samurais/hanlp-api/) [](https://hub.docker.com/r/samurais/hanlp-api/) [](https://microbadger.com/#/images/samurais/hanlp-api)
------
**HanLP**Java**HanLP**
**HanLP**
> *
*
* N-
* CRF
*
*
*
> *
> *
*
*
*
*
*
> *
* TextRank
> *
* TextRank
> *
*
> *
*
*
*
*
> *
*
*
> *
*
*
*
> *
*
* MaxEnt
* CRF
> *
*
*
* BiGram
*
* CoNLL
* CoNLL UA/LA/DA
**HanLP**
------
##
HanLPhttps://github.com/hankcs/HanLP/releases
SolrLucenehttps://github.com/hankcs/hanlp-solr-plugin
https://github.com/hankcs/HanLP/wiki
------
##
### Maven
Portablepom.xml
```
com.hankcs
hanlp
portable-1.3.5
```
CRFhanlp.properties
### jardatahanlp.properties
**HanLP**
#### 1jar
[hanlp.jar](https://github.com/hankcs/HanLP/releases)
#### 2data
| | | MB |
| -------- | -----: | :----: |
| [data.zip](https://github.com/hankcs/HanLP/releases) | | 255 |
HanLP
**HanLP**********
data
dictionary
model
model
-
- GitHubdata.zip
#### 3
:[hanlp.properties](https://github.com/hankcs/HanLP/releases)
GitHub```hanlp.properties``````jar``````zip```
HanLP
root=usr/home/HanLP/
data****data`/Users/hankcs/Documents/data``root=/Users/hankcs/Documents/`
- mini
```
CoreDictionaryPath=data/dictionary/CoreNatureDictionary.mini.txt
BiGramDictionaryPath=data/dictionary/CoreNatureDictionary.ngram.mini.txt
```
HanLP.propertiesclasspathsrcresourcesIDEclasspath
HanLP
##
**HanLP**`HanLP``HanLP.`IDE**HanLP**
*`HanLP`**HanLP***
Demo[com.hankcs.demo](https://github.com/hankcs/HanLP/tree/master/src/test/java/com/hankcs/demo)
### 1. Demo
```java
System.out.println(HanLP.segment("HanLP"));
```
-
* **HanLP****HanLP**
* **HanLP**
- **HanLP**
* Github**HanLP**data/**
* ****
* **TrieDAWGAhoCorasickDoubleArrayTrie
### 2.
```java
List termList = StandardTokenizer.segment("");
System.out.println(termList);
```
-
* **HanLP**`Tokenizer`
* `HanLP.segment``StandardTokenizer.segment`
* [HanLP](http://www.hankcs.com/nlp/part-of-speech-tagging.html#h2-8)
-
* [](http://www.hankcs.com/nlp/segment/the-word-graph-is-generated.html)
### 3. NLP
```java
List termList = NLPTokenizer.segment("");
System.out.println(termList);
```
-
* NLP`NLPTokenizer`
### 4.
```java
List termList = IndexTokenizer.segment("");
for (Term term : termList)
{
System.out.println(term + " [" + term.offset + ":" + (term.offset + term.word.length()) + "]");
}
```
-
* `IndexTokenizer``term.offset`
### 5. N-
```java
Segment nShortSegment = new NShortSegment().enableCustomDictionary(false).enablePlaceRecognize(true).enableOrganizationRecognize(true);
Segment shortestSegment = new DijkstraSegment().enableCustomDictionary(false).enablePlaceRecognize(true).enableOrganizationRecognize(true);
String[] testCase = new String[]{
",",
"",
};
for (String sentence : testCase)
{
System.out.println("N-" + nShortSegment.seg(sentence) + "\n" + shortestSegment.seg(sentence));
}
```
-
* N`NShortSegment`
* N
-
* [NJava](http://www.hankcs.com/nlp/segment/n-shortest-path-to-the-java-implementation-and-application-segmentation.html)
### 6. CRF
```java
Segment segment = new CRFSegment();
segment.enablePartOfSpeechTagging(true);
List termList = segment.seg("");
System.out.println(termList);
for (Term term : termList)
{
if (term.nature == null)
{
System.out.println("" + term.word);
}
}
```
-
* CRF
-
* [CRFJava](http://www.hankcs.com/nlp/segment/crf-segmentation-of-the-pure-java-implementation.html)
* [CRF++](http://www.hankcs.com/nlp/the-crf-model-format-description.html)
### 7.
```java
/**
* AhoCorasickDoubleArrayTrie
* @author hankcs
*/
public class DemoHighSpeedSegment
{
public static void main(String[] args)
{
String text = "";
System.out.println(SpeedTokenizer.segment(text));
long start = System.currentTimeMillis();
int pressure = 1000000;
for (int i = 0; i < pressure; ++i)
{
SpeedTokenizer.segment(text);
}
double costTime = (System.currentTimeMillis() - start) / (double)1000;
System.out.printf("%.2f", text.length() * pressure / costTime);
}
}
```
-
*
* i72000
-
* [Aho CorasickDoubleArrayTrie](http://www.hankcs.com/program/algorithm/aho-corasick-double-array-trie.html)
### 8.
```java
/**
*
*
* @author hankcs
*/
public class DemoCustomDictionary
{
public static void main(String[] args)
{
//
CustomDictionary.add("");
//
CustomDictionary.insert("", "nz 1024");
//
// CustomDictionary.remove("");
System.out.println(CustomDictionary.add("", "nz 1024 n 1"));
System.out.println(CustomDictionary.get(""));
String text = ""; //
// AhoCorasickDoubleArrayTrie
final char[] charArray = text.toCharArray();
CustomDictionary.parseText(charArray, new AhoCorasickDoubleArrayTrie.IHit()
{
@Override
public void hit(int begin, int end, CoreDictionary.Attribute value)
{
System.out.printf("[%d:%d]=%s %s\n", begin, end, new String(charArray, begin, end - begin), value);
}
});
// trie
BaseSearcher searcher = CustomDictionary.getSearcher(text);
Map.Entry entry;
while ((entry = searcher.next()) != null)
{
System.out.println(entry);
}
//
System.out.println(HanLP.segment(text));
}
}
```
-
* `CustomDictionary`
*
-
* `CustomDictionary``data/dictionary/custom/CustomDictionary.txt``CustomDictionaryPath=data/dictionary/custom/CustomDictionary.txt; .txt;`
*
-
* `[] [A] [A] [B] [B] ...`
* n`.txt ns;`
*
* ****
-
* [Trie](http://www.hankcs.com/program/java/tire-tree-participle.html)
* [Aho CorasickDoubleArrayTrie](http://www.hankcs.com/program/algorithm/aho-corasick-double-array-trie.html)
### 9.
```java
String[] testCase = new String[]{
"",
"",
"",
"",
"",
"",
",",
};
Segment segment = HanLP.newSegment().enableNameRecognize(true);
for (String sentence : testCase)
{
List termList = segment.seg(sentence);
System.out.println(termList);
}
```
-
* `HanLP.segment()`
* ```data/dictionary/person/nr.txt`` A 1`````
* pull request
-
* [HMM-Viterbi](http://www.hankcs.com/nlp/chinese-name-recognition-in-actual-hmm-viterbi-role-labeling.html)
### 10.
```java
String[] testCase = new String[]{
"Facebook",
"",
};
Segment segment = HanLP.newSegment().enableTranslatedNameRecognize(true);
for (String sentence : testCase)
{
List termList = segment.seg(sentence);
System.out.println(termList);
}
```
-
*
-
* [](http://www.hankcs.com/nlp/name-transliteration-cascaded-hidden-markov-model-and-japanese-personal-names-recognition.html)
### 11.
```java
String[] testCase = new String[]{
"3",
":",
};
Segment segment = HanLP.newSegment().enableJapaneseNameRecognize(true);
for (String sentence : testCase)
{
List termList = segment.seg(sentence);
System.out.println(termList);
}
```
-
*
-
* [](http://www.hankcs.com/nlp/name-transliteration-cascaded-hidden-markov-model-and-japanese-personal-names-recognition.html)
### 12.
```java
String[] testCase = new String[]{
"",
"",
};
Segment segment = HanLP.newSegment().enablePlaceRecognize(true);
for (String sentence : testCase)
{
List termList = segment.seg(sentence);
System.out.println(termList);
}
```
-
*
*
-
* [HMM-Viterbi](http://www.hankcs.com/nlp/ner/place-names-to-identify-actual-hmm-viterbi-role-labeling.html)
### 13.
```java
String[] testCase = new String[]{
"",
"",
"",
};
Segment segment = HanLP.newSegment().enableOrganizationRecognize(true);
for (String sentence : testCase)
{
List termList = segment.seg(sentence);
System.out.println(termList);
}
```
-
*
* HanLP
-
* [HMM-Viterbi](http://www.hankcs.com/nlp/ner/place-name-recognition-model-of-the-stacked-hmm-viterbi-role-labeling.html)
### 14.
```java
String content = "(Programmer)";
List keywordList = HanLP.extractKeyword(content, 5);
System.out.println(keywordList);
```
-
* `TextRankKeyword``TextRankKeyword.getKeywordList(document, size)`
-
* [TextRankJava](http://www.hankcs.com/nlp/textrank-algorithm-to-extract-the-keywords-java-implementation.html)
### 15.
```java
String document = "\n" +
"\n" +
"\n" +
"\n" +
"";
List sentenceList = HanLP.extractSummary(document, 3);
System.out.println(sentenceList);
```
-
* `TextRankSentence``TextRankSentence.getTopSentenceList(document, size)`
-
* [TextRankJava](http://www.hankcs.com/nlp/textrank-algorithm-java-implementation-of-automatic-abstract.html)
### 16.
```java
String text = "\n" +
"Algorithm\n" +
"\n" +
"1\n" +
"\n" +
"\n" +
"\n" +
"\n" +
"MATLAB\n" +
"\n" +
"2\n" +
" \n" +
"\n" +
"3\n" +
"/\n" +
"2D3D(2D-to-3D conversion)(de-interlacing)(Motion estimation/Motion Compensation)(Noise Reduction)(scaling)(Sharpness)(Super Resolution),(gesture recognition),(face recognition)\n" +
"RRMRTT\n" +
"\n" +
"";
List phraseList = HanLP.extractPhrase(text, 10);
System.out.println(phraseList);
```
-
* `MutualInformationEntropyPhraseExtractor``MutualInformationEntropyPhraseExtractor.extractPhrase(text, size)`
-
* [](http://www.hankcs.com/nlp/extraction-and-identification-of-mutual-information-about-the-phrase-based-on-information-entropy.html)
### 17.
```java
/**
*
* @author hankcs
*/
public class DemoPinyin
{
public static void main(String[] args)
{
String text = "";
List pinyinList = HanLP.convertToPinyinList(text);
System.out.print(",");
for (char c : text.toCharArray())
{
System.out.printf("%c,", c);
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin);
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getPinyinWithToneMark());
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getPinyinWithoutTone());
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getTone());
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getShengmu());
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getYunmu());
}
System.out.println();
System.out.print(",");
for (Pinyin pinyin : pinyinList)
{
System.out.printf("%s,", pinyin.getHead());
}
System.out.println();
}
}
```
-
* **HanLP**
* **HanLP**
* **HanLP**`AhoCorasickDoubleArrayTrie`
-
* [Java](http://www.hankcs.com/nlp/java-chinese-characters-to-pinyin-and-simplified-conversion-realization.html#h2-17)
### 18.
```java
/**
*
* @author hankcs
*/
public class DemoTraditionalChinese2SimplifiedChinese
{
public static void main(String[] args)
{
System.out.println(HanLP.convertToTraditionalChinese(""));
System.out.println(HanLP.convertToSimplifiedChinese(""));
}
}
```
-
* **HanLP**`=`**HanLP**
-
* [Java](http://www.hankcs.com/nlp/java-chinese-characters-to-pinyin-and-simplified-conversion-realization.html#h2-17)
### 19.
```java
/**
* ()
* @author hankcs
*/
public class DemoSuggester
{
public static void main(String[] args)
{
Suggester suggester = new Suggester();
String[] titleArray =
(
" \n" +
" \n" +
"\n" +
" \n" +
""
).split("\\n");
for (String title : titleArray)
{
suggester.addSentence(title);
}
System.out.println(suggester.suggest("", 1)); //
System.out.println(suggester.suggest("", 1)); //
System.out.println(suggester.suggest("mayun", 1)); //
}
}
```
-
* **HanLP**
*
### 20.
```java
/**
*
* @author hankcs
*/
public class DemoWordDistance
{
public static void main(String[] args)
{
String[] wordArray = new String[]
{
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
};
for (String a : wordArray)
{
for (String b : wordArray)
{
System.out.println(a + "\t" + b + "\t\t" + CoreSynonymDictionary.distance(a, b));
}
}
}
}
```
-
*
-
* IDIDID
### 21.
```java
/**
* CRF-Xms512m -Xmx512m -Xmn256mMaxEnt-Xms1g -Xmx1g -Xmn512m
* @author hankcs
*/
public class DemoDependencyParser
{
public static void main(String[] args)
{
CoNLLSentence sentence = HanLP.parseDependency("");
System.out.println(sentence);
//
for (CoNLLWord word : sentence)
{
System.out.printf("%s --(%s)--> %s\n", word.LEMMA, word.DEPREL, word.HEAD.LEMMA);
}
//
CoNLLWord[] wordArray = sentence.getWordArray();
for (int i = wordArray.length - 1; i >= 0; i--)
{
CoNLLWord word = wordArray[i];
System.out.printf("%s --(%s)--> %s\n", word.LEMMA, word.DEPREL, word.HEAD.LEMMA);
}
//
CoNLLWord head = wordArray[12];
while ((head = head.HEAD) != null)
{
if (head == CoNLLWord.ROOT) System.out.println(head.LEMMA);
else System.out.printf("%s --(%s)--> ", head.LEMMA, head.DEPREL);
}
}
}
```
-
* `NeuralNetworkDependencyParser``NeuralNetworkDependencyParser.compute(sentence)`
* `MaxEntDependencyParser.compute(sentence)`
-
* [](http://www.hankcs.com/nlp/parsing/neural-network-based-dependency-parser.html)
* [](http://www.hankcs.com/nlp/parsing/to-achieve-the-maximum-entropy-of-the-dependency-parser.html)
* [CRFJava](http://www.hankcs.com/nlp/parsing/crf-sequence-annotation-chinese-dependency-parser-implementation-based-on-java.html)
##
**HanLP****HanLP**
###
- `CoreNatureDictionary.txt`
* `[] [A] [A] [B] [B] ...`
*
* `.txt``,`****`.csv`Excel****
- `CoreNatureDictionary.ngram.txt`
* `[] []`
*
csv
###
Trie**HanLP**Trie
AhoCorasickDoubleArrayTrieTrie[Trie(DoubleArrayTrie)](http://www.hankcs.com/program/java/%E5%8F%8C%E6%95%B0%E7%BB%84trie%E6%A0%91doublearraytriejava%E5%AE%9E%E7%8E%B0.html)[ACAhoCorasickDoubleArrayTrie](http://www.hankcs.com/program/algorithm/aho-corasick-double-array-trie.html)
###
(filename.txt)(filename.txt.binfilename.txt.trie.datfilename.txt.trie.value)
-
* UTF-8CRLF
-
* .bin.trie.dat.trie.valuetrie
*
###
HanLP2014
```java
HanLP.Config.enableDebug();
```
-
* `data/dictionary/CoreNatureDictionary.txt`
* `CoreNatureDictionary.ngram.txt``CoreNatureDictionary.txt`
-
* `data/dictionary/CoreNatureDictionary.ngram.txt`
*
-
*
* [](http://www.hankcs.com/category/nlp/ner/)
pull request
------
##
###
- Apache License Version 2.0
- HanLP
- HanLPHanLP
###
- [darts-clone-java](https://github.com/hiroshi-manabe/darts-clone-java)
- [SharpICTCLAS](http://www.cnblogs.com/zhenyulu/archive/2007/04/18/718383.html)
- [snownlp](https://github.com/isnowfy/snownlp)
- [ansj_seg](https://github.com/NLPchina/ansj_seg)
- [nlp-lang](https://github.com/NLPchina/nlp-lang)
NLP
-
-
-
-
- An Efficient Implementation of Trie Structures, JUN-ICHI AOE AND KATSUSHI MORIMOTO
- TextRank: Bringing Order into Texts, Rada Mihalcea and Paul Tarau
HanLPHanLPNLP
[@hankcs](http://weibo.com/hankcs/)
20141216