Module org.apache.lucene.analysis.smartcn

Package org.apache.lucene.analysis.cn.smart

Class HMMChineseTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.util.SegmentingTokenizerBase

org.apache.lucene.analysis.cn.smart.HMMChineseTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class HMMChineseTokenizer
extends SegmentingTokenizerBase

Tokenizer for Chinese or mixed Chinese-English text.

The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
private final OffsetAttribute	offsetAtt
private static final BreakIterator	sentenceProto	used for breaking the text into sentences
private final CharTermAttribute	termAtt
private Iterator<SegToken>	tokens
private final TypeAttribute	typeAtt
private final WordSegmenter	wordSegmenter

Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

buffer, BUFFERMAX, offset

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
HMMChineseTokenizer()	Creates a new HMMChineseTokenizer
HMMChineseTokenizer(AttributeFactory factory)	Creates a new HMMChineseTokenizer, supplying the AttributeFactory

Method Summary

Modifier and Type	Method	Description
protected boolean	incrementWord()	Returns true if another word is available
void	reset()	This method is called by a consumer before it begins consumption using TokenStream.incrementToken().
protected void	setNextSentence(int sentenceStart, int sentenceEnd)	Provides the next input sentence for analysis

Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

end, incrementToken, isSafeEnd

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader, setReaderTestPoint

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

sentenceProto

private static final BreakIterator sentenceProto

used for breaking the text into sentences

termAtt

private final CharTermAttribute termAtt

offsetAtt

private final OffsetAttribute offsetAtt

typeAtt

private final TypeAttribute typeAtt

wordSegmenter

private final WordSegmenter wordSegmenter

tokens

private Iterator<SegToken> tokens

Constructor Details

HMMChineseTokenizer

public HMMChineseTokenizer()

Creates a new HMMChineseTokenizer

HMMChineseTokenizer

public HMMChineseTokenizer(AttributeFactory factory)

Creates a new HMMChineseTokenizer, supplying the AttributeFactory

Method Details

setNextSentence

protected void setNextSentence(int sentenceStart,
 int sentenceEnd)

Description copied from class: SegmentingTokenizerBase

Provides the next input sentence for analysis

Specified by:

setNextSentence in class SegmentingTokenizerBase

incrementWord

protected boolean incrementWord()

Description copied from class: SegmentingTokenizerBase

Returns true if another word is available

Specified by:

incrementWord in class SegmentingTokenizerBase

reset

public void reset()
           throws IOException

Description copied from class: TokenStream

This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

Overrides:

reset in class SegmentingTokenizerBase

Throws:

IOException