Processing chain, integration

GATE integration

We have integrated the different modules making up e-magyar.hu in the GATE language processing framework. One advantage of GATE, which is implemented in Java, is that it provides a convenient method for integrating any number of language processing tool (Processing Resource) in one system. Another of its advantages is a uniform annotation model, which enables the communication between the respective modules.

At the beginning of the processing all the spaces in the text are indexed with a number (so-called offset), and from there on every annotation is expressed by a pair of offsets, indicating the beginning and end of the annotation. Information is stored either directly in the annotation (Token), or in the attributes of the annotation (the attribute of the Token word stem). This way the different annotations do not interfere with each other; there can even be overlaps between them. This is a useful solution: every module may read only the annotation(s) relevant for it, while the output can be written in the existing or newly created annotation. For example: the tokeniser creates Token and SpaceToken units, in accordance with words and spaces; the morphological analyser will only fetch the list of Tokens, running the morphological analysis on these and ignoring the SpaceTokens. The modules can be parameterised with respect to which annotations they should be working with, which increases the flexibility of the system even more.

Our task is, then, to make every module capable of treating both its input and output according to the GATE annotation model. An additional task is, if we would like to specify a relation between independent annotations, it must be done explicitly. An obvious example for this is the relation between proper names and the tokens constituting them. Such tasks have been implemented during the integration.

Modules in the processing chain

The toolchain e-magyar.hu has the following modules integrated in GATE: emToken segments a text into sentences and tokens, emMorph carries out a morphological analysis and determines possible word stems, emTag disambiguates, i.e. choses the valid morphological analysis and lemma from the possible ones. emDep and emCons carries out syntactic parsing, followed by an additional tool connecting verbs and their respective separable prefixes, returning the prefixed verb stem. Finally, emChunk determines noun phrases, while emNer identifies proper names. These later tools add an IOB annotation to a given attribute, which, for a more convenient further processing, is transformed into an independent annotation by an additional tool.

Installation

Use in GATE Developer

After installing the Lang_Hungarian GATE plugin, which contains the toolchain e-magyar.hu, we should carry out the following steps:

1. Loading the processing tools: rightclick on Processing Resources in the left panel, and choosing the required tools.
2. Creating a new Corpus Pipeline in the Applications section of the left panel.
3. Clicking on the newly created Corpus Pipeline and putting together the processing chain by arranging the chosen tools in the list on the right side, following the required order. One should put a Document Reset PR at the top of the list, which will reset the document in its default state before each run. This can be loaded from the ANNIE plugin, which is always at our disposal.
4. Creating a Language Resource in the left panel: a new GATE Document, which will contain the text to be processed.
5. Creating a corpus from the text: rightclick on the newly created GATE Document, and New Corpus with this Document.
6. Clicking on the Corpus Pipeline, then specifying the newly created corpus in the middle of the screen, at Corpus, then clicking on the Run this Application button.

The results can be viewed by clicking on the newly created GATE Document, by switching on the Annotation Sets and the Annotation List. By placing the mouse over the respective units, their annotation becomes visible.

For further details and possibilities of GATE, its documentation should be consulted.

For developers