What is Natural Language Processing (NLP)?

German terms for Natural Language Processing (NLP) are computational linguistics or linguistic data processing. NLP is a subfield of artificial intelligence. NLP deals with theoretical foundations and practical methods and techniques to mechanically capture, analyze, understand, process and generate natural language (spoken or in text form) using computers and algorithms. Computational linguistics works in an interdisciplinary manner and applies knowledge and methods from computer science, linguistics and data science. The aim of computational linguistics is to enable people to communicate comprehensively with computers and to support them in various tasks and problems. The complexity and ambiguity of natural language poses numerous challenges for natural language processing.

The sub-areas of Natural Language Processing NLU and NLG

Natural Language Processing can be divided into the two sub-areas Natural Language Understanding (NLU) and Natural Language Generation (NLG). Natural Language Understanding deals with understanding natural language. Grammar, syntax and semantics of natural language words and sentences are recorded and analyzed. The aim is to identify the sense and meaning of a text. Natural Language Generation aims to generate text in natural language.

Basic functionality of Natural Language Processing

In order to understand or generate human language, the unstructured data of natural language is converted into a structured form that can be processed by computers. Statistical approaches, algorithms and methods such as machine learning are used to capture syntax and semantics. Various steps are carried out before the meaning of a text is fully understood. Speech or text is first broken down into its individual components and analyzed. In this context, NLP tasks such as tokenization, part-of-speech tagging, lemmatization, stemming, parsing, literal disambiguation or named entity recognition have to be mastered. With the so-called sentiment analysis, it is even possible to recognize the emotional attitude and positive or negative attitudes of an author. Once a text has been entered, the information can be further processed automatically and, for example, answers or solutions for specific questions and problems can be generated. The answers and solutions are generated in text form and can be converted into acoustic speech output via text-to-speech conversion.

Many NLP tasks are handled with the help of machine learning. Machine learning models are trained on large amounts of data. Thanks to their trained “knowledge”, they are then able to process the texts presented to them in the desired form. However, stylistic devices such as irony, rhetorical questions, sarcasm or paradoxes pose problems even for very mature and “intelligent” ML models.

Practical applications for NLP

There are numerous practical applications for natural language processing, such as:

• summarizing texts

• Information extraction from text-based sources

• machine translations

• automatic spelling and grammar checker

• automatic indexing of literature

• Automated creation of texts (e.g. product descriptions)

• automated analysis of scientific or medical texts

• Analysis, categorization and answering of customer inquiries
• Detection of spam or phishing messages