Related papers
Flesch-Kincaid is Not a Text Simplification Evaluation Metric
Teerapaun Tanprasert
2021
Sentence-level text simplification is currently evaluated using both automated metrics and human evaluation. For automatic evaluation, a combination of metrics is usually employed to evaluate different aspects of the simplification. Flesch-Kincaid Grade Level (FKGL) is one metric that has been regularly used to measure the readability of system output. In this paper, we argue that FKGL should not be used to evaluate text simplification systems. We provide experimental analyses on recent system output showing that the FKGL score can easily be manipulated to improve the score dramatically with only minor impact on other automated metrics (BLEU and SARI). Instead of using FKGL, we suggest that the component statistics, along with others, be used for posthoc analysis to understand system behavior.
View PDFchevron_right
One Step Closer to Automatic Evaluation of Text Simplification Systems
Ruslan Mitkov
Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 2014
This study explores the possibility of replacing the costly and time-consuming human evaluation of the grammaticality and meaning preservation of the output of text simplification (TS) systems with some automatic measures. The focus is on six widely used machine translation (MT) evaluation metrics and their correlation with human judgements of grammaticality and meaning preservation in text snippets. As the results show a significant correlation between them, we go further and try to classify simplified sentences into: (1) those which are acceptable; (2) those which need minimal post-editing; and (3) those which should be discarded. The preliminary results, reported in this paper, are promising.
View PDFchevron_right
The C-Score -- Proposing a Reading Comprehension Metrics as a Common Evaluation Measure for Text Simplification
Irina Temnikova
View PDFchevron_right
CLaC @ QATS: Quality Assessment for Text Simplification
Leila Kosseim
ArXiv, 2017
This paper describes our approach to the 2016 QATS quality assessment shared task. We trained three independent Random Forest classifiers in order to assess the quality of the simplified texts in terms of grammaticality, meaning preservation and simplicity. We used the language model of Google-Ngram as feature to predict the grammaticality. Meaning preservation is predicted using two complementary approaches based on word embedding and WordNet synonyms. A wider range of features including TF-IDF, sentence length and frequency of cue phrases are used to evaluate the simplicity aspect. Overall, the accuracy of the system ranges from 33.33% for the overall aspect to 58.73% for grammaticality.
View PDFchevron_right
Language-independent Metric for Measuring Text Simplification that does not Require a Parallel Corpus
Lucas Mucida
Proceedings of the ... International Florida Artificial Intelligence Research Society Conference, 2022
Natural language processing encompasses several tasks, one of which is the automatic text simplification. Telling whether one text is simpler than another involves not only knowledge about the language being analyzed, but also a cultural knowledge of the target audience to which the text is being directed. Most of the current metrics used to measure text simplification are based on the use of parallel corpora, prepared by humans, which makes it difficult to apply the metrics in automatic text simplification in real time. In this paper, we present ISiM (Independent Simplification Metric), a metric that dismiss a parallel corpus, is simple, fast, language and human annotation independent, capable of quantifying the simplicity/complexity of a sentence, thus contributing improve automating text simplification. The results of the tests performed indicate that the proposed metric has the potential to be used to evaluate automatic methods of simplification.
View PDFchevron_right
Readability assessment for text simplification
Sandra Maria Aluísio
We describe a readability assessment ap-proach to support the process of text simplifi-cation for poor literacy readers. Given an in-put text, the goal is to predict its readability level, which corresponds to the literacy level that is expected from the target reader: rudi-mentary, basic or advanced. We complement features traditionally used for readability as-sessment with a number of new features, and experiment with alternative ways to model this problem using machine learning methods, namely classification, regression and ranking. The best resulting model is embedded in an authoring tool for Text Simplification.
View PDFchevron_right
Controllable Sentence Simplification
Benoît Sagot
2020
Text simplification aims at making a text easier to read and understand by simplifying grammar and structure while keeping the underlying information identical. It is often considered an all-purpose generic task where the same simplification is suitable for all; however multiple audiences can benefit from simplified text in different ways. We adapt a discrete parametrization mechanism that provides explicit control on simplification systems based on Sequence-to-Sequence models. As a result, users can condition the simplifications returned by a model on attributes such as length, amount of paraphrasing, lexical complexity and syntactic complexity. We also show that carefully chosen values of these attributes allow out-of-the-box Sequence-to-Sequence models to outperform their standard counterparts on simplification benchmarks. Our model, which we call ACCESS (as shorthand for AudienCe-CEntric Sentence Simplification), establishes the state of the art at 41.87 SARI on the WikiLarge te...
View PDFchevron_right
Automatic Text Simplification of News Articles in the Context of Public Broadcasting
Michel Simard
arXiv (Cornell University), 2022
This report summarizes the work carried out by the authors during the Twelfth Montreal Industrial Problem Solving Workshop, held at Université de Montréal in August 2022. The team tackled a problem submitted by CBC/Radio-Canada on the theme of Automatic Text Simplification (ATS). In order to make its written content more widely accessible, and to support its second-language teaching activities, CBC/RC has recently been exploring the potential of automatic methods to simplify texts. They have developed a modular lexical simplification system (LSS), which identifies complex words in French and English texts, and replaces them with simpler, more common equivalents. Recently however, the ATS research community has proposed a number of approaches that rely on deep learning methods to perform more elaborate transformations, not limited to just lexical substitutions, but covering syntactic restructuring and conceptual simplifications as well. The main goal of CBC/RC's participation in the workshop was to examine these new methods and to compare their performance to that of their own LSS. This report is structured as follows: In Section 2, we detail the context of the proposed problem and the requirements of the sponsor. We then give an overview of current ATS methods in Section 3. Section 4 provides information about the relevant datasets available, both for training and testing ATS methods. As is often the case in natural language processing applications, there is much less data available to support ATS in French than in English; therefore, we also discuss in that section the possibility of automatically translating English resources into French, as a means of supplementing the French data. The outcome of text simplification, whether automatic or not, is notoriously difficult to evaluate objectively; in Section 5, we discuss the various evaluation methods we have considered, both manual and automatic. Finally, we present the ATS methods we have tested and the outcome of their evaluation in Section 6, then Section 7 concludes this document and presents research directions.
View PDFchevron_right
Automatic text simplification is an NLP task that has received growing attention in rece
Stefan Bott
2013
Lexical simplification is the task of replacing a word in a given context by an easier-to-understand synonym. Although a number of lexical simplification approaches have been developed in recent years, most of them have been applied to English, with recent work taking advantage of parallel monolingual datasets for training. Here we present LexSiS, a lexical simplification system for Spanish that does not require a parallel corpus, but instead relies on freely available resources, such as an on-line dictionary and the Web as a corpus. LexSiS uses three techniques for finding a suitable word substitute: a word vector model, word frequency, and word length. In experiments with human informants, we have verified that LexSiS performs better than a hard-to-beat baseline based on synonym frequency.
View PDFchevron_right
Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish
Simon Mille, Horacio Saggion
ACM Transactions on Accessible Computing (TACCESS) - Special Issue on Speech and Language Processing for AT (Part 2), 2015
The way in which a text is written can be a barrier for many people. Automatic text simplification is a natural language processing technology that, when mature, could be used to produce texts that are adapted to the specific needs of particular users. Most research in the area of automatic text simplification has dealt with the English language. In this article, we present results from the Simplext project, which is dedicated to automatic text simplification for Spanish. We present a modular system with dedicated procedures for syntactic and lexical simplification that are grounded on the analysis of a corpus manually simplified for people with special needs. We carried out an automatic evaluation of the system's output, taking into account the interaction between three different modules dedicated to different simplification aspects. One evaluation is based on readability metrics for Spanish and shows that the system is able to reduce the lexical and syntactic complexity of the texts. We also show, by means of a human evaluation, that sentence meaning is preserved in most cases. Our results, even if our work represents the first automatic text simplification system for Spanish that addresses different linguistic aspects, are comparable to the state of the art in English Automatic Text Simplification.
View PDFchevron_right