by Rock Brentwood <markwh04@[EMAIL PROTECTED]
>
Apr 25, 2008 at 12:24 PM
On Apr 8, 12:44 pm, Tegiri Nena**** <TegiriNena...@[EMAIL PROTECTED]
> wrote:
> Formally, a set of terminals is partitioned into a separator or a set
> of separators, and the rest of terminals. Then, string tokenizer
> translates a given word into a set (or list) of words. Here we have
> the first technical difficulty, what exactly this translation is?
The inverse of a monoid homomorphism. The homomorphism it is the
inverse of is the one that maps the monoid of token sequences to the
monoid of character sequences.
