Applied Sciences Special Issue : Natural Language Processing: Trends and Challenges

In fact, MT/NLP research almost died in 1966 according to the ALPAC report, which concluded that MT is going nowhere. But later, some MT production systems were providing output to their customers (Hutchins, 1986) [60]. By this time, work on the use of computers for literary and linguistic studies had also started. As early as 1960, signature work influenced by AI began, with the BASEBALL Q-A systems (Green et al., 1961) [51]. LUNAR (Woods,1978) [152] and Winograd SHRDLU were natural successors of these systems, but they were seen as stepped-up sophistication, in terms of their linguistic and their task processing capabilities. There was a widespread belief that progress could only be made on the two sides, one is ARPA Speech Understanding Research (SUR) project (Lea, 1980) and other in some major system developments projects building database front ends.

This work is supported in part by the National Basic Research Program of China (973 Program, 2014CB340301).

The context of a text may include the references of other sentences of the same document, which influence the understanding of the text and the background knowledge of the reader or speaker, which gives a meaning to the concepts expressed in that text. Semantic analysis focuses on literal meaning of the words, but pragmatic analysis focuses on the inferred meaning that the readers perceive based on their background knowledge. ” is interpreted to “Asking for the current time” in semantic analysis whereas in pragmatic analysis, the same sentence may refer to “expressing resentment to someone who missed the due time” in pragmatic analysis. Thus, semantic analysis is the study of the relationship between various linguistic utterances and their meanings, but pragmatic analysis is the study of context which influences our understanding of linguistic expressions.

Share this article

It takes the information of which words are used in a document irrespective of number of words and order. In second model, a document is generated by choosing a set of word occurrences and arranging them in any order. This model is called multi-nomial model, in addition to the Multi-variate Bernoulli model, it also captures information on how many times a word is used in a document. Most text categorization approaches to anti-spam Email filtering have used multi variate Bernoulli model (Androutsopoulos et al., 2000) [5] [15].

Give this NLP sentiment analyzer a spin to see how NLP automatically understands and analyzes sentiments in text (Positive, Neutral, Negative). Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive

positive feedback from the reviewers. Imbalanced sample sizes in health literacy estimations were a major concern for developing the LP algorithm. For instance, compared to what we observed with respect to self-reported HL and what is known from the HL literature, there were relatively fewer people than expected who were modeled to have low LP and more people than expected to have high LP. The traditional ML algorithms such as Support Vector Machine (SVM) and Linear Discriminant Analysis (LDA) do not work well with imbalanced or skewed data.

In this paper, we first distinguish four phases by discussing different levels of NLP and components of Natural Language Generation followed by presenting the history and evolution of NLP. We then discuss in detail the state of the art presenting the various applications of NLP, current trends, and challenges. Finally, we present a discussion on some available datasets, models, and evaluation metrics in NLP. Hiring an investigator with expertise in methodologic evaluation who had no prior exposure to the research was beyond the scope of our project. However, since this paper was conceptualized post hoc, four years into the ECLIPPSE Project, the documents that were reviewed were not developed, organized, or preserved in such a way as to systematically enable a recount of challenges and solutions. Thus, some documents may not have fully reflected all relevant challenges in the study.

• Natural Language Processing is a subfield of Artificial Intelligence capable of breaking down human language and feeding the tenets of the same to the intelligent models.
• Finally, we present a discussion on some available datasets, models, and evaluation metrics in NLP.
• As they grow and strengthen, we may have solutions to some of these challenges in the near future.
• Like the culture-specific parlance, certain businesses use highly technical and vertical-specific terminologies that might not agree with a standard NLP-powered model.

There are complex tasks in natural language processing, which may not be easily realized with deep learning alone. It involves language understanding, language generation, dialogue management, knowledge base access and inference. Dialogue management can be formalized as a sequential decision process and reinforcement learning can play a critical role.

Article Access Statistics

We first give insights on some of the mentioned tools and relevant work done before moving to the broad applications of NLP. In addition to these, UIU has taken a pioneering step by introducing a dedicated data science program, making it the first university in Bangladesh to offer such a specialized curriculum. With NLP spending expected to increase in 2023, now is the time to understand how to get the greatest value for your investment. For many businesses, the chatbot is a primary communication channel on the company website or app. It’s a way to provide always-on customer support, especially for frequently asked questions.

There are a multitude of languages with different sentence structure and grammar. Machine Translation is generally translating phrases from one language to another with the help of a statistical engine like Google Translate. The challenge with machine translation technologies is not directly translating words but keeping the meaning of sentences intact along with grammar and tenses. In recent years, various methods have been proposed to automatically evaluate machine translation quality by comparing hypothesis translations with reference translations. In this research paper, a comprehensive literature review was undertaken in order to analyze Natural Language Processing (NLP) application based in different domains. Also, by conducting qualitative research, we will try to analyze the development of the current state and the challenge of NLP technology as a key for Artificial Intelligence (AI) technology, pointing out some of the limitations, risks and opportunities.

Natural Language Processing (NLP) Challenges

This Special Issue aims to provide a comprehensive overview of the current trends, emerging technologies, and persistent challenges in NLP. It seeks to highlight the cutting-edge developments and address the hurdles the NLP community faces in this dynamic field. Emotion detection investigates and identifies the types of emotion from speech, facial expressions, gestures, and text. Sharma (2016) [124] analyzed the conversations in Hinglish means mix of English and Hindi languages and identified the usage patterns of PoS. Their work was based on identification of language and POS tagging of mixed script. They tried to detect emotions in mixed script by relating machine learning and human knowledge.

In addition to describing the painstaking nature of their work, they noted the limitation of applying a single index to one portal system, which limits both its robustness as well as its translation to other clinical web portals. We recognized this challenge in our own work and attempted to address it by using NLP to broaden the diversity of lexical and syntactic indices combined with machine learning techniques to predict LP and CP. However, faced with hundreds of indices related to literacy and text difficulty, we employed standard statistical methods to reduce the number of indices combined with empirically and theoretically motivated decisions. Employing a greater diversity of linguistic tools and features, while enhancing processing efficiency and comprehensiveness, created different analytic challenges (e.g., word/character processing limits, skewed results, etc.). Finding workable solutions was critical to moving forward and was a direct result of different ideas and approaches emerging from our interdisciplinary collaboration. As most of the world is online, the task of making data accessible and available to all is a challenge.

Evaluation metrics and challenges

By doing so, we achieved a more balanced proportion of SMs that met the threshold versus those that were below threshold for both the LP and CP algorithms. These computational processes and their validity are detailed in papers that describe the development of our LP2 and CP5 algorithms. We also explored the extent to which alternative ML approaches (such as under-sampling, oversampling or SMOTE) that correct for imbalanced data might be more appropriate. Ultimately, we decided in favor of adjusting the thresholds, but plan to explore alternative techniques in future research. Several approaches were used to reduce the number of linguistic indices included in the LP and CP algorithms. First, we reduced the set by applying typical filtering methods such as removing indices based on multi-collinearity, non-normal distributions, and non-normal variance (e.g., zero or near zero variance).

An HMM is a system where a shifting takes place between several states, generating feasible output symbols with each switch. The sets of viable states and unique symbols may be large, but finite and known. Few of the problems could be solved by Inference A certain sequence of output symbols, compute the probabilities of one or more candidate states with sequences. Patterns matching the state-switch sequence are most likely to have generated a particular output-symbol sequence. Training the output-symbol chain data, reckon the state-switch/output probabilities that fit this data best. The objective of this section is to present the various datasets used in NLP and some state-of-the-art models in NLP.

This work was also supported by “The Health Delivery Systems Center for Diabetes Translational Research (CDTR),” funded by the NIH/NIDDK (P30DK092924). The content is solely the responsibility of the authors and does not necessarily represent the official views of NLM, NIDDK, AHRQ or the NIH. We would like to acknowledge Dr. Aaron Likens and Dr. Jianmin Dai for the programming work they did for the ECLIPPSE study. In order to address the minimum word requirement for processing some of the NLP algorithms, we applied a threshold wherein an SM could not contain fewer than 50 words for NLP analysis for patient secure messages. The objective of this section is to discuss evaluation metrics used to evaluate the model’s performance and involved challenges. Seunghak et al. [158] designed a Memory-Augmented-Machine-Comprehension-Network (MAMCN) to handle dependencies faced in reading comprehension.

A major challenge was application of linguistic tools (available at linguisticanalysistools.org) for extracting and selecting the indices used to train the machine learning models for the LP and CP algorithms. The indices were selected from linguistic tools that export hundreds of indices applied to the SMs exchanged between the patients and the physicians [28,30]. As such, various decisions needed to be made regarding whether and how to reduce the set of indices. Hidden Markov Models are extensively used for speech recognition, where the output sequence is matched to the sequence of individual phonemes. HMM is not restricted to this application; it has several others such as bioinformatics problems, for example, multiple sequence alignment [128].

We also identified that physicians’ SMs occasionally included what appeared to be automated content. Often known as “smart texts” or “smart phrases,” these reflect standardized stock content that physicians can use by selecting from a menu of pre-determined responses (see Table 1). NLU enables machines to understand natural language and analyze it by extracting concepts, entities, emotion, keywords etc. It is used in customer care applications to understand the problems reported by customers either verbally or in writing. Linguistics is the science which involves the meaning of language, language context and various forms of the language. So, it is important to understand various important terminologies of NLP and different levels of NLP.

Data Availability Statement

This is clearly an advantage compared to the traditional approach of statistical machine translation, in which feature engineering is crucial. We think that, among the advantages, end-to-end training and representation learning really differentiate deep learning from traditional machine learning approaches, and make it powerful machinery for natural language processing. In our view, there are five major tasks in natural language processing, namely classification, matching, translation, structured prediction and the sequential decision process. Most of the problems in natural language processing can be formalized as these five tasks, as summarized in Table 1.

For example, in the sentences “he is going to the riverbank for a walk” and “he is going to the bank to withdraw some money”, word2vec will have one vector representation for “bank” in both the sentences whereas BERT will have different vector representation for “bank”. Muller et al. [90] used the BERT model to analyze the tweets on covid-19 content. The use of the BERT model in the legal domain was explored by Chalkidis et al. [20]. Using these approaches is better as classifier is learned from training data rather than making by hand. The naïve bayes is preferred because of its performance despite its simplicity (Lewis, 1998) [67] In Text Categorization two types of models have been used (McCallum and Nigam, 1998) [77]. But in first model a document is generated by first choosing a subset of vocabulary and then using the selected words any number of times, at least once irrespective of order.

Obviously, combination of deep learning and reinforcement learning could be potentially useful for the task, which is beyond deep learning itself. Working with experts across several scientific disciplines also presented unique challenges (Table 1). For instance, similar terms often have different meanings across health services research, clinical epidemiology, cognitive science, and linguistics. This extended into defini tional differences related to tasks and methods delegated and employed, resulting in some confusion and inefficiency. This led to debates about the value of creating two separate indices vs one common index to allow comparison between patient HL and physician CP on the same scale. Another critical trans-disciplinary related to different interpretations of the real-world significance of certain findings, and concerns about research integrity or rigor.

According to Spring wise, Waverly Labs’ Pilot can already transliterate five spoken languages, English, French, Italian, Portuguese, and Spanish, and seven written affixed languages, German, Hindi, Russian, Japanese, Arabic, Korean and Mandarin Chinese. The Pilot earpiece is connected via Bluetooth to the Pilot speech translation app, which uses speech recognition, machine translation and machine learning and speech synthesis technology. Simultaneously, the user will hear the translated version of the speech on the second earpiece. Moreover, it is not necessary that conversation would be taking place between two people; only the users can join in and discuss as a group. As if now the user may experience a few second lag interpolated the speech and translation, which Waverly Labs pursue to reduce.

• They can respond to your questions via their connected knowledge bases and some can even execute tasks on connected “smart” devices.
• Manuscripts should be submitted online at by registering and logging in to this website.
• Expert.ai’s NLP platform gives publishers and content producers the power to automate important categorization and metadata information through the use of tagging, creating a more engaging and personalized experience for readers.
• We attempted to develop a novel, automated measure of readability of health-related text that was generated from computational linguistic analyses of physicians’ written language [6].
• Where a search engine returns results that are sourced and verifiable, ChatGPT does not cite sources and may even return information that is made up—i.e., hallucinations.

Challenges and solutions matrix for patient literacy profile (LP) and physician complexity profile (CP). You can foun additiona information about ai customer service and artificial intelligence and NLP. There is a system called MITA (Metlife’s Intelligent Text Analyzer) (Glasgow et al. (1998) [48]) that extracts information from life insurance applications. Ahonen et al. (1998) [1] suggested a mainstream framework for text mining that uses pragmatic and discourse level analyses of text.

Sonnhammer mentioned that Pfam holds multiple alignments and hidden Markov model-based profiles (HMM-profiles) of entire protein domains. The cue of domain boundaries, family members and alignment are done semi-automatically found on expert knowledge, sequence similarity, other protein family databases and the capability of HMM-profiles to correctly identify and align the members. HMM may be used for a variety of NLP applications, including word prediction, sentence production, quality assurance, and intrusion detection systems [133]. There are challenges of deep learning that are more common, such as lack of theoretical foundation, lack of interpretability of model, and requirement of a large amount of data and powerful computing resources.

The front-end projects (Hendrix et al., 1978) [55] were intended to go beyond LUNAR in interfacing the large databases. In early 1980s computational grammar theory became a very active area of research linked with logics for meaning and knowledge’s ability to deal with the user’s beliefs and intentions and with functions like emphasis and themes. It is often possible to perform end-to-end training in deep learning for an application. This is because the model (deep neural network) offers rich representability and information in the data can be effectively ‘encoded’ in the model. For example, in neural machine translation, the model is completely automatically constructed from a parallel corpus and usually no human intervention is needed.

The choice of area in NLP using Naïve Bayes Classifiers could be in usual tasks such as segmentation and translation but it is also explored in unusual areas like segmentation for infant learning and identifying documents for opinions and facts. Anggraeni et al. (2019) [61] used ML and AI to create a question-and-answer system for retrieving information about hearing loss. They developed I-Chat Bot which understands the user input and provides an appropriate response and produces a model which can be used in the search for information about required hearing impairments. The problem with naïve bayes is that we may end up with zero probabilities when we meet words in the test data for a certain class that are not present in the training data. The NLP and ML strategies developed in ECLIPPSE have yielded novel high-throughput measures that can assess components of patient HL and physician linguistic complexity by analyzing written (email) messages exchanged between patients and their healthcare providers [1,2,4]. In our effort to create a generally applicable and accurate set of tools, we tested multiple linguistic analysis tools and strategies.

This was helpful in mitigating the tensions between the theoretical vs. applied aspects of the project. A language can be defined as a set of rules or set of symbols where symbols are combined and used for conveying information or broadcasting the information. Since all the users may not be well-versed in machine specific language, Natural Language Processing (NLP) caters those users who do not have enough time to learn new languages or get perfection in it. In fact, NLP is a tract of Artificial Intelligence and Linguistics, devoted to make computers understand the statements or words written in human languages.

The Robot uses AI techniques to automatically analyze documents and other types of data in any business system which is subject to GDPR rules. It allows users to search, retrieve, flag, classify, and report on data, mediated to be super sensitive under GDPR quickly and easily. Users also can identify personal data from documents, view feeds on the latest personal data that requires attention and provide reports on the data suggested to be deleted natural language processing challenges or secured. RAVN’s GDPR Robot is also able to hasten requests for information (Data Subject Access Requests – “DSAR”) in a simple and efficient way, removing the need for a physical approach to these requests which tends to be very labor thorough. Peter Wallqvist, CSO at RAVN Systems commented, “GDPR compliance is of universal paramountcy as it will be exploited by any organization that controls and processes data concerning EU citizens.

As far as categorization is concerned, ambiguities can be segregated as Syntactic (meaning-based), Lexical (word-based), and Semantic (context-based). Simply put, NLP breaks down the language complexities, presents the same to machines as data sets to take reference from, and also extracts the intent and context to develop them further. For the unversed, NLP is a subfield of Artificial Intelligence capable of breaking down human language and feeding the tenets of the same to the intelligent models.

Text analysis models may still occasionally make mistakes, but the more relevant training data they receive, the better they will be able to understand synonyms. Luong et al. [70] used neural machine translation on the WMT14 dataset and performed translation of English text to French text. The model demonstrated a significant improvement of up to 2.8 bi-lingual evaluation understudy (BLEU) scores compared to various neural machine translation systems.

Deciding on a sub-sample with which to assess expert-rated HL and expert-rated physician linguistic complexity and determining thresholds for them both based on SM content presented additional challenges. In the recent past, models dealing with Visual Commonsense Reasoning [31] and NLP have also been getting attention of the several researchers and seems a promising and challenging area to work upon. These models try to extract the information from an image, video using a visual reasoning paradigm such as the humans can infer from a given image, video beyond what is visually obvious, such as objects’ functions, people’s intents, and mental states. Wiese et al. [150] introduced a deep learning approach based on domain adaptation techniques for handling biomedical question answering tasks. Their model revealed the state-of-the-art performance on biomedical question answers, and the model outperformed the state-of-the-art methods in domains. The world’s first smart earpiece Pilot will soon be transcribed over 15 languages.

All these forms the situation, while selecting subset of propositions that speaker has. The information that populates an average Google search results page has been labeled—this helps make it findable by search engines. However, the text documents, reports, PDFs and intranet pages that make up enterprise content are unstructured data, and, importantly, not labeled. This makes it difficult, if not impossible, for the information to be retrieved by search.

Too many results of little relevance is almost as unhelpful as no results at all. As a Gartner survey pointed out, workers who are unaware of important information can make the wrong decisions. Proposed by Google AI Research, Bidirectional Encoder Representations from Transformers (BERT) is a State of the Art (SOTA) model in Natural Language Processing (NLP).

Unique concepts in each abstract are extracted using Meta Map and their pair-wise co-occurrence are determined. Then the information is used to construct a network graph of concept co-occurrence that is further analyzed to identify content for the new conceptual model. Medication adherence is the most studied drug therapy problem and co-occurred with concepts related to patient-centered interventions targeting self-management. The framework requires additional refinement and evaluation to determine its relevance and applicability across a broad audience including underserved settings.

With the recent focus on large language models (LLMs), AI technology in the language domain, which includes NLP, is now benefiting similarly. You may not realize it, but there are countless real-world examples of NLP techniques that impact our everyday lives. Even humans at times find it hard to understand the subtle differences in usage. Therefore, despite NLP being considered one of the more reliable options to train machines in the language-specific domain, words with similar spellings, sounds, and pronunciations can throw the context off rather significantly.

Publishers and information service providers can suggest content to ensure that users see the topics, documents or products that are most relevant to them. Now, thanks to AI and NLP, algorithms can be trained on text in different languages, making it possible to produce the equivalent meaning in another language. This technology even extends to languages like Russian and Chinese, which are traditionally more difficult to translate due to their different alphabet structure and use of characters instead of letters. Despite being one of the more sought-after technologies, NLP comes with the following rooted and implementation AI challenges. Manuscripts should be submitted online at by registering and logging in to this website. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website.