Textual Data Selection based on Mean Square Difference Probability for Language Modeling

مزوج فريحة; بن يطو عبدالقادر

Article Details

Textual Data Selection based on Mean Square Difference Probability for Language Modeling

Authors:

مزوج فريحة | بن يطو عبدالقادر

Publisher:

Université Ibn Khaldoun de Tiaret

Journal:

Models & Optimisation and Mathematical Analysis Journal Vol.6 Issue 1, pp.10-14

Document Type:

Article

Publication Date:

12/2018

Document Language:

English

Document ID (Org No.):

2001404098

pISSN:

2588-2112

Keywords:

Abstract:

 The language model (LM) is an important module in many applications that produce natural language text such as Automatic Speech Recognition, Machine Translation systems, etc. Generally, the amount of training data which are suitable for training language models dedicated to specific target task is limited. Hence this kind of textual data are too costly to produce, the use of textual data selected from others domains can be useful. This paper proposes to investigate the Mean Square Difference Probability (MSDP) criteria between two models representing respectively in-domain and out-domain-specific data for textual data selection. This technique is analyzed and tested on French broadcast news and TV shows transcription data. Results show that, the selection data based on Mean Square Difference Probability is competitive compared to other criteria of state of the art such as Difference Cross-Entropy (dXent) data selection.

Article Details

Textual Data Selection based on Mean Square Difference Probability for Language Modeling

Authors:

مزوج فريحة | بن يطو عبدالقادر

Publisher:

Université Ibn Khaldoun de Tiaret

Journal:

Models & Optimisation and Mathematical Analysis Journal Vol.6 Issue 1, pp.10-14

Document Type:

Article

Publication Date:

12/2018

Document Language:

English

Document ID (Org No.):

2001404098

pISSN:

2588-2112

cross-entropy

data selection

mean square difference probability

n-gram language model

perplexity

textual corpus

Issues

Article Details

Textual Data Selection based on Mean Square Difference Probability for Language Modeling

Authors: مزوج فريحة | بن يطو عبدالقادر

Publisher: Université Ibn Khaldoun de Tiaret

Journal: Models & Optimisation and Mathematical Analysis Journal Vol.6 Issue 1, pp.10-14

Document Type: Article

Publication Date: 12/2018

Document Language: English

Document ID (Org No.): 2001404098

pISSN: 2588-2112

cross-entropy

data selection

mean square difference probability

n-gram language model

perplexity

textual corpus

Issues

Authors:

مزوج فريحة | بن يطو عبدالقادر

Publisher:

Université Ibn Khaldoun de Tiaret

Journal:

Models & Optimisation and Mathematical Analysis Journal Vol.6 Issue 1, pp.10-14

Document Type:

Article

Publication Date:

12/2018

Document Language:

English

Document ID (Org No.):

2001404098

pISSN:

2588-2112