Text
segmentation

From Wikipedia, the free encyclopedia

Jump to: navigation, search

This article needs
additional citations for verification.
Please help improve
this article by adding
citations to reliable sources. Unsourced material may be challenged
and removed. (October
2011)

Text segmentation is the process of dividing written text into meaningful
units, such as words,
sentences, or topics. The term
applies both to mental processes used by
humans when reading text, and to artificial processes implemented in computers, which are the
subject of natural language
processing. The problem is non-trivial, because while some written languages
have explicit word boundary markers, such as the word spaces of written English and the
distinctive initial, medial and final letter shapes of Arabic, such signals are
sometimes ambiguous and not present in all written languages.

Compare speech segmentation,
the process of dividing speech into linguistically meaningful portions.

Contents

1 Segmentation problems

1.1 Word
segmentation

1.2 Sentence
segmentation

1.3 Text
segmentation

1.4 Other segmentation
problems

2 Automatic segmentation
approaches

3 See also

4 References

5 External
links

Segmentation
problems[edit]

Word segmentation[edit]

See also: Word > Word
boundary

Word segmentation is the problem of dividing a string of written language
into its component words.

In English and many other
languages using some form of the Latin alphabet, the space is a good
approximation of a word divider (word delimiter).
(Some examples where the space character alone may not be sufficient include
contractions like can't for can not.)

However the equivalent to this character is not found in all written scripts,
and without it word segmentation is a difficult problem. Languages which do not
have a trivial word segmentation process include Chinese, Japanese, where sentences but
not words are delimited, Thai and Lao,
where phrases and sentences but not words are delimited, and Vietnamese, where
syllables but not words are delimited.

In some writing systems however, such as the Ge'ez script used for Amharic and Tigrinya among other languages,
words are explicitly delimited (at least historically) with a non-whitespace
character.

The Unicode Consortium
has published a Standard Annex on
Text Segmentation, exploring the issues of segmentation in multiscript
texts.

Word splitting is the process of parsing concatenated text (i.e.
text that contains no spaces or other word separators) to infer where word
breaks exist.

Word splitting may also refer to the process of hyphenation.

Sentence
segmentation[edit]

See also: Sentence
boundary disambiguation

Sentence segmentation is the problem of dividing a string of written language
into its component sentences. In English and some
other languages, using punctuation, particularly the full stop character is a
reasonable approximation. However even in English this problem is not trivial
due to the use of the full stop character for abbreviations, which may or may
not also terminate a sentence. For example Mr. is not its own sentence in
"Mr. Smith went to the shops in Jones Street." When processing plain
text, tables of abbreviations that contain periods can help prevent incorrect
assignment of sentence boundaries.

As with word segmentation, not all written languages contain punctuation
characters which are useful for approximating sentence boundaries.

Text segmentation[edit]

Main articles: Topic
analysis and Document
classification

Topic analysis consists of two main tasks: topic identiﬁcation and text
segmentation. While the first is a simple classification of a
specific text, the latter case implies that a document may contain multiple
topics, and the task of computerized text segmentation may be to discover these
topics automatically and segment the text accordingly. The topic boundaries may
be apparent from section titles and paragraphs. In other cases, one needs to use
techniques similar to those used in document
classification.

Segmenting the text into topics or discourse
turns might be useful in some natural processing tasks: it can improve
information retrieval or speech recognition significantly (by
indexing/recognizing documents more precisely or by giving the specific part of
a document corresponding to the query as a result). It is also needed in Topic
detection and Tracking systems and text summarizing
problems.

Many different approaches have been tried:[1][2]
e.g. HMM, lexical
chains, passage similarity using word co-occurrence, clustering etc.

It is quite an ambiguous task — people evaluating the text segmentation
systems often differ in topic boundaries. Hence, evaluating is quite dubious
problem too.

Other segmentation
problems[edit]

Processes may be required to segment text into segments besides mentioned,
including morphemes (a task usually
called morphological
analysis) or paragraphs.

Automatic
segmentation approaches[edit]

Automatic segmentation is the problem in natural language
processing of implementing a computer process to segment text.

When punctuation and similar clues are not consistently available, the
segmentation task often requires fairly non-trivial techniques, such as
statistical decision-making, large dictionaries, as well as consideration of
syntactic and semantic constraints. Effective natural language processing
systems and text segmentation tools usually operate on text in specific domains
and sources. As an example, processing text used in medical records is a very
different problem than processing news articles or real estate
advertisements.

The process of developing text segmentation tools starts with collecting a
large corpus of text in an application domain. There are two general
approaches:

Manual analysis of text and writing custom software

Annotate the sample corpus with boundary information and use Machine
Learning

Some text segmentation systems take advantage of any markup like HTML and
know document formats like PDF to provide additional evidence for sentence and
paragraph boundaries.

See also[edit]

Hyphenation

Natural
language processing

Speech
segmentation

Lexical analysis

Word
count

v

t

e

Natural
Language Processing

General Terms

Text
corpus

Speech
corpus

Stopwords

Bag-of-words

AI-complete

n-gram (Bigram, Trigrams)

Text analysis

Text segmentation

POS
Tagging

Text
Chunking

Compound
term processing

Stemming

NER

Coreference
resolution

Sentiment
analysis

Concept
mining

Parsing

Word
sense disambiguation

Terminology
extraction

Truecasing

Automatic
summarization

Multi-document
summarization

Sentence
extraction

Text
simplification

Machine
Translation

Computer-assisted

Example
Based

Rule
Based

AIDC

Speech
recognition

Speech
synthesis

OCR

Natural
language generation

Topic model

Pachinko
allocation

LDA

LSA

Computer-assisted
reviewing

Grammar
checker

Automated
essay scoring

Concordancer

Natural
language user interface

Question
answering

Chatterbot

Automated
online
assistant

References[edit]

^
Freddy Y. Y. Choi
(2000). "Proceedings of
the 1st Meeting of the North American Chapter of the Association for
Computational Linguistics (ANLP-NAACL-00)" (PDF).
pp. 26–33.  |chapter=
ignored (help)

^
Jeffrey
C. Reynar (1998). "Topic Segmentation: Algorithms and Applications" (PDF).
IRCS-98-21. University of
Pennsylvania. Retrieved 2007-11-08.

External links[edit]

Word Segment An open source software tool for word
segmentation in Chinese.

Word Split An open source software tool designed to split
conjoined words into human-readable text.

Stanford
Segmenter An open source software tool for word segmentation in Chinese or
morpheme segmentation in Arabic.

KyTea An open source software tool for word segmentation in
Japanese and Chinese.

Chinese Notes A Chinese-English dictionary that also does
word segmentation.

Zhihuita Segmentor A high precision and high performance
Chinese segmentation freeware.

Retrieved from "http://en.wikipedia.org/w/index.php?title=Text_segmentation&oldid=631132937"

Categories:

Tasks
of natural language processing

Hidden categories:

CS1
errors: Chapter ignored

Articles
needing additional references from October 2011

All
articles needing additional references

Navigation menu

Personal tools

Create
account

Log
in

Namespaces

Article

Talk

Variants

Views

Read

Edit

View
history

More

Search

Navigation

Main page

Contents

Featured
content

Current
events

Random article

Donate
to Wikipedia

Wikimedia Shop

Interaction

Help

About Wikipedia

Community
portal

Recent
changes

Contact
page

Tools

What
links here

Related
changes

Upload
file

Special
pages

Permanent
link

Page
information

Wikidata item

Cite
this page

Print/export

Create
a book

Download
as PDF

Printable
version

Languages

বাংলা

Deutsch

Galego

Հայերեն

Bahasa
Indonesia

日本語

Norsk
nynorsk

Português

Русский

Edit
links

This page was last modified on 26 October 2014, at
02:13.

Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to
the Terms of
Use and Privacy Policy.
Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a
non-profit organization.

Privacy
policy

About Wikipedia

Disclaimers

Contact
Wikipedia

Developers

Mobile
view