A Machine Learning Approach for Cross Script Named Entity Recognition

computer engineering

An essential information extraction subtask is named entity recognition (NER). Multiword phrases with specific meaning, such as those referring to people, places, or organisations, are recognised and categorised. Most of the time, these expressions convey the text's main ideas. Better document structuring and text filtering can be done using this information. It can be a source of data for additional natural language processing (NLP) operations like question answering, summarization, or machine translation. The NER framework as it is now has two main problems. The system must be calibrated for each new language or domain, which is the first problem. When a framework made for one space is used in another, the quality of the result suffers significantly. Even more challenging is the change from one language to another. The second problem is the lack of external and semantic information, which is important for people to recognise names in texts like posts on online forums. Using a variety of machine learning algorithms, including the Naive Bayes Classifier, Support Vector Machine, Random Forest Classifier, and Conditional Random Filed, this paper describes the development of the NER framework for the Wikipedia dataset crawled based on coarse NE Indian Cross Script context (list of person, location, organisation, and miscellaneous). The framework makes use of a variety of attributes that aid in the prediction of various named entities (NEs). Language dependent as well as language independent features are included in the set of features employed in this work. We created a dataset of 2916 course NEs from the Cross script Roman Hindi Wikipedia article and tagged them with a set of four different NE classes. Only the labels for Person names, Location names, Organization names, and Miscellaneous were counted. The 584 NE course token sets have been used to test the framework. The accuracy and F1 measurement of the performance are assessed. The Naive Bayes Classifier, Support Vector Machine Classifier, Random Forest, and Conditional Random Filed Classifier all produce F1-measures of 0.75 for Person name, Location name, and Organization name, 0.76 for Location name, 0.78 for Organization name, and 0.85 for Location name. When applying the Naive Bayes Classifier, the accuracy for Person name, Location name, and Organization name is seen to be 78%, 80%, and 81% respectively.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.