Large scale concept dictionaries such as WordNet (Miller et al., 1993; Fellbaum, 1998), EuroWordNet (Vossen, 1998), FrameNet (Baker & Fillmore, 1998), and EDR (EDR, 1996) have existed since the 1990s. The first three were constructed by psychologists or linguists. WordNet is an English electronic concept dictionary that was created at Princeton University. It organizes English words and phrases into synonym sets (i.e., synsets) representing the underlying lexical concepts. The synsets are linked with each other via their relationships, such as super-sub and antonym. EuroWordNet, which was created in the same way as WordNet, is a multilingual database of several European languages. FrameNet is based on the Frame Semantics by C. Fillmore at Berkeley, which used the Frame semantics (case frames) of verbs. The last one, EDR, is a Japanese concept dictionary and was created by computer engineers for machine translations between Japanese and English.

Our proposed Japanese concept dictionary, which is based on the results of large-scale human association experiments, is a model of the hierarchical structure of human concepts and the semantic relations among concepts. In comparison with WordNet and EDR it was found to have reasonable structures. Our concept dictionary contains new associative relations features due to the quantitative distances among concepts calculated using the linear programming method. This dictionary has good potential for use in semantic analysis in natural language processing (NLP) tasks. The conventional concept dictionaries calculate distances using the number of links between concepts along its hierarchical structure. However, since the concepts of daily life have fine sub-categorization, the number of links between them increases and, as a result, the distances become too long. Conventional dictionaries have few applications for systems of natural language understanding because the distances among the concepts in the dictionaries are difficult to measure. It is, therefore, difficult to show their usability in high level applications. In this paper, our dictionary is applied to text summarization and word sense disambiguation systems in the ways described in the following sections.

Text summarization methods generally require deep semantic processing and background knowledge in order to approach the level of human results. Much of the previous work has used superficial clues (Watanabe, 1996) and ad hoc heuristics. In summarizing texts, the word frequency approach or connectionist approach (Hashida et al., 1987) have often been used for calculating the importance scores of words; the importance scores of sentences are then calculated as the sum of those of the words in the sentences (Mochizuki & Okumura, 2000) (Zechner, 1996). Summarization based on a concept dictionary such as WordNet uses a semantic graph-based representation for the documents (Plaza, 2009).

The Contextual Semantic Network is used in our research to calculate the importance scores for sentences given in the input document. The results are compared with those from human summarization experiments and those from conventional methods using word frequencies. The comparison shows that the system does well in summarization tasks.

Word sense disambiguation is one of the most difficult problems in the field of NLP because it needs contextual meanings. A lot of previous work on such disambiguation used the co-occurrence of words in context. Several machine learning algorithms, such as the Naive Bayes method or the Support Vector Machine (Murata et al., 2003), are based on such co-occurrence. The effectiveness of neural network approaches to word sense disambiguation has been suggested (Waltz & Pollack, 1985). Not only neural network architecture but also large-scale machine-readable dictionaries for word sense disambiguation have been exploited (Veronis & Ide, 1990). For knowledge-driven advanced language processing, the WordNet glosses are disambiguated by increasing the connectivity of WordNet concepts (Moldovan, 2004).

Human-beings can assign an appropriate word sense to anambiguous word in a sentence based on the words that follow it. We propose a Dynamic Contextual Network Model for word sense disambiguation, where the Contextual Semantic Network has a structure that dynamically changes depending on each input word sequentially in an input sentence. In this model, the network architecture is based on the proposed concept dictionary, which includes the semantic relations among concepts/words, and the relations can be represented using the quantitative distances between them. In the word sense disambiguation process, an interactive activation model is used to identify a word’s meaning on the Contextual Semantic Network.

Background knowledge is crucial for computers to better understand contextual information as well as the syntactic or shallow semantic information from texts. The Associative Concept Dictionary (hereafter ACD) was created based on the results of large-scale online association experiments in which many participants simultaneously used the campus network at Shonan Fujisawa Campus of Keio University. The details of the ACD were described in the Journal of NLP (Okamoto & Ishizaki, 2001), which was written in Japanese, and the essential parts of the paper are described below.

In the experiments, the stimulus words that were used were fundamental nouns used in Japanese elementary school textbooks in which homonyms were excluded. Fifty participants focused on each stimulus word. In the experiment, a participant was requested to associate from ten stimulus words with a given set of semantic relations, such as hypernym, hyponym, part/material, attribute, synonym, action, and situation, and to input the associated words using a Japanese input system.

All of the associated words in the ACD have distances to the stimulus words calculated by using the following linear programming method. The distance D(x,y) between concepts, x and y, is shown by means of the following equation (Okamoto & Ishizaki, 2001):

where F(x, y) = N_x/(n + δ), δ = N_x/10 – 1, (N_x ≥ 10) and s(x, y) =

s_xyi. N_x denotes the number of participants who received the stimulus word x, and n_xy denotes the number of participants who inputted the associated word y for a stimulus word x with a semantic relation. Furthermore, δ denotes a factor introduced to limit the maximum value of F(x, y) to 10, and s_xyi denotes an order of the associated word y by a participant i for a given stimulus word x.

The ACD is built using quantified distances and is organized in a hierarchical structure in terms of hypernyms and hyponyms. The attribute information is used to explain the features of a given word. It also includes the action and situation concepts related to the stimulus words. It contains 1100 stimulus words and the total number of the associated words is about 280,000. It has about 64,000 different associated words when the overlapping words are not counted. The experiments have been going on for more than ten years. The total number of participants for the experiments has been increasing every year and is currently more than 5000 people. Figure 1 shows a description for the stimulus word “魚:fish” in the ACD. The second column represents the semantic relations given to the participants. The third column shows the associated words for “fish”. The numerals in <1> represent the frequencies of participants who gave the same associated word. The numerals in <2> are the averages of the participants’ association order, where 1.0 is the first place of the association. The numerals in <3> expressed the calculated conceptual distances using equation (1). We have distributed at no cost a CD-ROM containing the ACD to more than fifty research organizations who sent email requests to Prof. Ishizaki, one of the authors and agreed to a simple contract.

We compared the associative concept dictionary with EDR and WordNet using the distance information described in our dictionary (Okamoto & Ishizaki, 2001). Several familiar words in daily human life were chosen to compare the three dictionaries. These words, for example, tree, car, train, and fruit, have a relatively large number of associative words. We calculated the distances between those words and their hypernyms in these three dictionaries. The results of the principal component analysis show that the conceptual structure of ACD is more similar to that of WordNet than that of EDR.

Text summarization has conventionally been accomplished by extracting important sentences from a document based on various superficial cues. For example, in such conventional methods, the frequency of the occurrence of a given word in a document has often been used in calculating the importance scores of sentences. In this research, the Contextual Semantic Network (hereafter CSN) is developed using the ACD. A spreading activation model is used to calculate a word’s score on the CSN, where the activation values on the network are calculated using quantitative distances among the concepts.

3.1.1 Important sentence extraction using CSN

In the proposed summarization method, the CSN is used to calculate the importance scores of the sentences in the input document. One cannot use only the information obtained from the word co-occurrence inits context, but must also use that from a comparatively rich network with quantitative distances and contextual information for extracting important sentences.

The following steps are the procedural details for the CNS construction according to an input document for summarization tasks (Figure 2). These steps are used again for word sense disambiguation in Section 4.1.1.

For example, let the input sentences be “ガラパゴスにはゾウガメがい る．その亀は島の中を歩いている: There is a giant tortoise in Galapagos. This turtle is walking around the island.” In this text, “ゾウガメ: A giant tortoise” is a hyponym of “亀: a turtle”. The importance score of “a turtle” is calculated using the distance between “turtle” and “giant tortoise” in the CSN. “ガラパゴス: Galapagos” is an island and has a situation relation with “ゾウガメ: a giant tortoise”. “歩く: Walk” is a verb concept of “a giant tortoise” and also that of “a turtle”. We can construct an intra-document network since all the words are included in the ACD. In addition, some hypernym words are added in the CSN such as “生物: living-thing”, which is a hypernym of “a turtle.”

The activation value of each node is calculated using a spreading activation method on the CSN. The initial values (V_a(0)) of the words in sentence k of the input document are calculated using equation (2) below. Next, the activation value of each node N_a is calculated by means of equation (3).

where the calculation is repeated until Σ_a=1 – (V_a (t) – V_a (t + 1))² reaches a certain small value. The decay parameter θ is assumed to be 0.1 based on our test experiments. V_a (t) is the activation value of node N_a at time t. S(j, N_a) is the number of nodes (word) N_a appearing in document j.D_abD_ab is the distance between two concepts. V_b (t) is the activation value of the node connected to node N_a. α is a normalized value where the total number of links is divided by the maximum distance. P_jkl is the score of words (N_a) in sentence k in document j. For each important sentence score (T_jk), the sum of the keywords-weights is divided by the number of words (L_jk), as shown in the following equation:

3.1.2 Extracting important sentences based on word frequency

We use the following important sentence extraction methods to compare them with our proposed methods. The input texts are morphologically analyzed using the Japanese dependency structure analysis software Cabocha. The importance scores are calculated using the root morphemes of the nouns, adjectives, adverbs, and verbs. Pronouns and certain nouns (number, counter suffix, and so on) are not included in the calculations.

In this method, the word w_jkl is defined as the l^th word in the k^th sentence in document j. The importance score P_jkl of word w_jkl is calculated using the following equation (5),

where F_jkl is the frequency of word w_jkl in document j, N is the total number of documents, and n_jkl is the number of documents where word w_jkl appears. The sentence scores (T_jk) are calculated using equation (4).

3.1.3 Important sentence extraction by human participants

We used eight documents extracted from Japanese elementary school textbooks because the ACD is constructed of the basic nouns in these textbooks. These documents contain about 17 sentences on average (range 10-23). All the documents focus on a single topic in natural science and consist of titles and documents.

We carried out an experiment with forty participants all of whom were native Japanese speakers and students from Keio University. We requested the participants to choose the five most important sentences from the document and to arrange them in the order of their importance in the document. The importance scores were from 5 to 1. An importance score of 5 is given to the most important sentence and a score of 1 to the fifth most important sentence. Next, the important sentences are sorted on the basis of the sum of the importance scores given to each sentence by all the participants.

Next, we calculated the degree of coincidence using Kendall’s coefficients of concordance (W) among the participants’ results of extracting important sentences. Kendall’s coefficients of concordance show that there was a degree of agreement among the sentences ordered by the participants. A high W value means that the ordering results by the participants were consistent with each other. Table 1 shows Kendall’s coefficients of concordance (W) and the numbers of sentences in the eight documents.

Kendall’s coefficients of concordance (W) of three of the documents (D3, D4, and D7) were relatively lower than the others. However, the chi-square analysis indicates that the difference is not statistically significant.

3.2.1 Evaluation of our method and conventional methods

In order to show the effectiveness of the ranks of the sentences obtained by our system, we compared our results with those extracted by the participants and those automatically obtained using the conventional methods based on the word frequencies. First, importance scores from 10 to 6 were given respectively to the top five sentences that the participants chose. For a sentence chosen both by the participants and by one of the other methods, the value of correspondence (C) is calculated using the following equation (Okamoto & Ishizaki, 2003).

where R_i (p, m) is the importance score (range 10-6) of the top five sentences that the participants extracted, and Δr_i (p, m) is a value obtained by subtracting the rank of a sentence extracted by the participants from the rank of the sentence extracted by the system. The above equation compares the ranks using the computational methods (our method and the word frequency method) with those by the participants. For example, if sentences (15, 10, 2, 6, 8) are the human extraction results and (15, 8, 10, 14, 5) are the computational ones, the value of correspondence is calculated using equation (6), such as C = 10 + (9 – 1) + (6 – 3) = 21. The correspondence of extracted sentences (CS) is the number of same sentence extractions between the human and computational method results. In the previous example, CS was 3.

Table 2 lists a comparison of our method with the word frequency methods. The data in the “Our method” rows in Table 2 was obtained by comparing the top five sentences that our method chose using CSN with those chosen by the participants. The data in the “Word frequency method” rows in Table 2 was obtained by comparing the top five sentences using the word frequencies and those chosen by the participants. The CS in the table is the number of sentences from the top five sentences extracted using the automatic methods that correspond to the sentences extracted by the participants. In Table 2, the C for our method was high. The results show that this method, using the CSN, was more effective than the word frequency method. The Cs of some documents (D3, D4, and D7) are relatively lower than the ones from other documents. These results correspond with the results of Kendall’s coefficients of concordance (W), which were also relatively low (see Table 1).

Numerous Japanese ideographs have several different meanings and pronunciations. For example, the ideograph “額,” which has two pronunciations, /hitai/ and /gaku/, in which the former represents “the forehead of the human body” and the latter “the amount of money” or “a frame”. In the association experiments, such ideographs were given as stimulus words followed by their pronunciations to avoid ambiguities.

A Dynamic Contextual Network Model (hereafter DCNM) disambiguates word senses by means of a spreading activation model in the CSN. In addition, this network model is not static but dynamic, where the network structure changes depending on the context of the words in the sentences. In the proposed model, when the network is not yet rich enough to disambiguate word senses and when the model cannot decide the appropriate meaning of the homographic ideograms, it will get another input word from the input sentence sequentially and add it to the network to enhance it. Using the dynamic model we can assign appropriate word senses to ambiguous words by comparing the activation values and choosing the best one for the homographic ideogram meanings.

4.1.1 Construction of CSN for homographic ideograph

The CSN is constructed starting from the ambiguous words and a dependency structure of the input text. Then, concepts associated from the words are added to the CNS using the semantic relations and the quantitative distances in the ACD.The CSN is constructed using the ACD for word sense disambiguation. The following steps (C and D) are added to steps A and B for the CSN construction using the content words in Section 3.1.1.

Figure 3 shows an example of a CSN. The square shape nodes are the input words in the sentence for the two meanings of the homographic ideographs and are connected with an inhibitory link. The oval shape nodes are added using the ACD and are connected with the excitatory links. The dotted lines are the inhibitory links. The associated word nodes of a homographic ideograph are connected with other homographic ideographs using the inhibitory links. “Museum” is a situational concept of “frame”. “Picture” and “face” do not exist in the sentence but are obtained from the ACD.

4.1.2 Activation value calculation in the Network

The activation value of each node is calculated based on the interactive activation model (McCelland & Rumelhart, 1981) on the CSN. We define the maximum activation level as 1.0. An initial value (a_i(0)) and each initial value (a_i(t)) for the sequential input words are calculated using the following equations.

where 1.0 is the normalization value for S_ki, which is the number of nodes N_i that appear in sentence k. Next, the new activation value (a_i (t + 1)) of each node Ni at time t + 1 is calculated using the following equation.

where the decay parameter θ is assumed to be 0.1 and ε_i (t) expresses the influence of its neighbors at time t. When the neighbors of a node are active, they affect the activation value of the node by their excitatory or inhibitory connections, depending on the link between the two nodes. Those excitatory and inhibitory influences are combined using a simple equation (9) to yield a net input to the node. Thus, n_i (t) represents the net input to the node using the following equation.

where a_j (t) denotes the activation value of node N_j connected with node Ni, α is a constant weight, given by the total number of links of the CSN, and D_ij denotes the distance between two nodes, N_i and N_j. In this article, the CSN is constructed by tracing the semantic relation paths while accumulating the distance until the value of 5.0 is exceeded. Therefore, the value of D_ij with an inhibitory link is assumed to be -5.0. When the net input is excitatory, n_i (t) > 0, the effect on the node, ε_i (t), is given by the following equation.

where M is the maximum activation level of the node and is set to 1.0.When the net input is inhibitory, n_i (t) ≤ 0, the effect of the input on the node is given by the following equation.

where m is the minimum activation level of the node and is set to 0.

4.1.3 Enhancement of the Contextual Semantic Network

Human beings try to assign appropriate word senses to a given ambiguous word in a sentence using the contextual information located before the word; when they cannot disambiguate the word, they will use the words that follow the ambiguous word. In our model, if there is only a slight difference between the activation values of the two nodes for a given input word, it enhances the network by adding the words that follow it from the input sentence. Figure 4 shows an example of enhancing CSN depending on the words that follow it.

Let us take as input the following Japanese sentence: “壁のピカソの絵の額 が落ちて，頭に当たって額から血が出た,” and its translation in English: “The frame of Picasso’s picture dropped from the wall, struck my head, and my forehead bled.”

In this sentence, “frame” and “forehead” are English expressions that correspond to the homographic ideograph “額” in Japanese. The input word order follows the Japanese order. At first, we construct the CSN based on the ACD for the first input word “wall”. Then, the nouns in the sentence, including the ambiguous words, are added sequentially to the network and all the nodes activate each other by the method outlined in Section 4.1.2.

Figure 5 shows the activation values of the homographic ideographs and the other input words in the simulation, where each input word spans twenty time cycles. The horizontal axis represents time, and the vertical axis represents the activation values. The words in the rectangles are the major words selected from the input sentence. For the homographic ideogram, “frame” has a bigger value than that of “forehead” (see broken-line circle in Figure. 5) in the first half of the sentence, so we can assign the appropriate pronunciation and meaning to “frame” as that from the Japanese ideograph “額.” However, since the homographic ideogram “forehead” has a bigger value than that of “frame” (see thick-line circle in Figure. 5) in the last half of the sentence, in this case we can assign “forehead” to the Japanese ideograph “額.”

In this article we have proposed a simulation model for word sense disambiguation and an application system for document summarization. Both use contextual information and the quantitative distances among the concepts in the ACD. The method for summarization shows that it is better than the conventional ones and the method for disambiguation shows that it can dynamically identify the meaning of ambiguous words according to the input words.

The ACD is currently a small size dictionary compared with other concept dictionaries. We will extend it to a large-scale dictionary by automatically extracting concepts from the corpora. This extension will be useful for higher level contextual understanding systems.

The evaluation of the word sense disambiguation method using DCNM is a future project from the NLP point of view. As a preliminary evaluation, we used the multinomial Naive Bayes text classification method. We have obtained success rates of roughly 90% for both methods. We need to increase the test data to conduct a more precise evaluation.

Human word sense disambiguation processes have been observed using MEG (Magnetoencephalogram) (Ihara et al., 2008). We also have observed such brain activities using multi-channel Near Infrared Spectroscopy (NIRS). These studies will be useful for enhancing our DCNM as a model for human language understanding as well as computer systems for NLP. NIRS is expected to be useful for the real-time analysis of human brain activity during language processing.