A Machine Learning Approach for Cross Script Named Entity Recognition -A REVIEW

computer engineering

Named Entity Recognition (NER) is an important subtask of information extraction. It recognizes and classifies multiword expressions with particular meaning, e.g. persons, locations, organizations etc. Most of the time, these expressions carry the core information of the text. This information can be utilized for better structuring of documents, filtering of essential texts. It can be used as an input for other natural language processing (NLP) tasks like question answering, summarization or machine translation. There are two fundamental issues of current NER framework. The first issue is the necessity to calibrate the system each new language or domain. There is extensive fall in the quality of the output, when a framework is intended for one space is utilized for another one. Transition from one language to another language is even more complicated. Second issue is the absence of external and semantic knowledge, which is significant for individuals to perceive names in texts such as internet forum posts. This paper reports about the development of NER framework for Wikipedia dataset crawled based on Cross Script coarse NE Indian context (list of person, location, organization and miscellaneous) by using various machine learning algorithms like Naïve Bayes Classifier, Support Vector Machine, Random Forest Classifier and Conditional Random Filed. The framework uses various types of features that are helpful in predicting different named entities (NEs). The set of features used for this work includes language dependent as well as language independent components. We have built the dataset of 2916 course NEs from Wikipedia page for Cross script Roman Hindi labeled with a label set of four diverse NE classes. We accounted just the labels that signify Person names, Location names, Organization names and Miscellaneous. The framework has been tested with the course token sets of 584 NEs. The performance is evaluated in terms of F1-measaure and accuracy. The F1-measure for Person name, Location name, Organization name is observed as 0.75 using Naïve Bayes Classifier, 0.76 using Support vector Machine Classifier, 0.78 using Random Forest and 0.85 using Conditional Random Filed Classifier. The accuracy for Person name, Location name, Organization name is observed as 78% using Naïve Bayes Classifier, 80% using Support vector Machine Classifier, 81% using Random Forest and 87% using Conditional Random Filed Classifier. By this, we conclude that the Conditional Random Field gives the best F1-measure and accuracy on Wikipedia NER dataset.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.