Data Preprocessing

Summary

Data preprocessing can refer to manipulation or dropping of data before it is used in order to ensure or enhance performance, and is an important step in the data mining process. The phrase "garbage in, garbage out" is particularly applicable to data mining and machine learning projects. Data collection methods are often loosely controlled, resulting in out-of-range values, impossible data combinations, and missing values, amongst other issues. Analyzing data that has not been carefully screened for such problems can produce misleading results. Thus, representation and quality of data is necessary before running any analysis. Often, data preprocessing is the most important phase of a machine learning project, especially in computational biology. If there is a high proportion of irrelevant and redundant information present or noisy and unreliable data, then knowledge discovery during the training phase may be more difficult. Data preparation and filtering steps can take a considerable

Official source

https://en.wikipedia.org/wiki/Data_Preprocessing

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (91)

Related publications

Related people (5)

Related people

Related units (4)

Related units

Related concepts (5)

Related concepts

Related courses (24)

Related courses

Related lectures (47)

Related lectures

Automatic Segmentation of Nuclei in Histopathology Images using Encoding-Decoding Convolutional Neural Networks

Beril Besbinar, Pascal Frossard, Deniz Sayin Mercadier

Accurate and fast segmentation of nuclei in histopathological images plays a crucial role in cancer research for detection and grading, as well as personal treatment. Despite the important efforts, current algorithms are still suboptimal in terms of speed, adaptivity and generalizability. Popular Deep Convolutional Neural Networks (DCNNs) have recently been utilized for nuclei segmentation, outperforming traditional approaches that exploit color and texture features in combination with shallow classifiers or segmentation algorithms. However, DCNNs need large annotated datasets that require extensive amount of time and expert knowledge. In addition, segmentation results obtained by either traditional or DCNN approaches often require a post-processing step to separate cluttered nuclei. In this paper, we propose a computationally efficient nuclei segmentation framework based on DCNNs exhibiting an encoding-decoding structure. We use a partially-annotated dataset and develop an effective training solution. We also use a weighted background model for network to give more importance to borders of nuclei to overcome the problem of clutters. The abolition of any pre-processing or post-processing step without any compromise on the performance leads to a fast and parameter-free system, which presents important advantages with respect to state-of-the-art.

IEEE2019

Unsupervised Voice Activity Detection by Modeling Source and System Information using Zero Frequency Filtering

Mathew Magimai Doss, Eklavya Sarkar

Voice activity detection (VAD) is an important pre-processing step for speech technology applications. The task consists of deriving segment boundaries of audio signals which contain voicing information. In recent years, it has been shown that voice source and vocal tract system information can be extracted using zero-frequency filtering (ZFF) without making any explicit model assumptions about the speech signal. This paper investigates the potential of zero-frequency filtering for jointly modeling voice source and vocal tract system information, and proposes two approaches for VAD. The first approach demarcates voiced regions using a composite signal composed of different zero-frequency filtered signals. The second approach feeds the composite signal as input to the rVAD algorithm. These approaches are compared with other supervised and unsupervised VAD methods in the literature, and are evaluated on the Aurora-2 database, across a range of SNRs (20 to -5 dB). Our studies show that the proposed ZFF-based methods perform comparable to state-of-art VAD methods and are more invariant to added degradation and different channel characteristics.

2022

Real world applications such as hands-free dialling in cars may have to perform recognition of spoken digits in potentially very noisy environments. Existing state-of-the-art solutions to this problem use feature-based Hidden Markov Models (HMMs), with a preprocessing stage to clean the noisy signal. However, the effect that the noise has on the induced HMM features is difficult to model exactly and limits the performance of the HMM system. An alternative to feature-based HMMs is to model the clean speech waveform directly, which has the potential advantage that including an explicit model of additive noise is straightforward. One of the most simple model of the clean speech waveform is the autoregressive (AR) process. Being too simple to cope with the nonlinearity of the speech signal, the AR process is generally embedded into a more elaborate model, such as the Switching Autoregressive HMM (SAR-HMM). In this thesis, we extend the SAR-HMM to jointly model the clean speech waveform and additive Gaussian white noise. This is achieved by using a Switching Linear Dynamical System (SLDS) whose internal dynamics is autoregressive. On an isolated digit recognition task where utterances have been corrupted by additive Gaussian white noise, the proposed SLDS outperforms a state-of-the-art HMM system. For more natural noise sources, at low signal to noise ratios (SNRs), it is also significantly more accurate than a feature-based HMM system. Inferring the clean waveform from the observed noisy signal with a SLDS is formally intractable, resulting in many approximation strategies in the literature. In this thesis, we present the Expectation Correction (EC) approximation. The algorithm has excellent numerical performance compared to a wide range of competing techniques, and provides a stable and accurate linear-time approximation which scales well to long time series such as those found in acoustic modelling. A fundamental issue faced by models based on AR processes is that they are sensitive to variations in the amplitude of the signal. One way to overcome this limitation is to use Gain Adaptation (GA) to adjust the amplitude by maximising the likelihood of the observed signal. However, adjusting model parameters without constraint may lead to overfitting when the models are sufficiently flexible. In this thesis, we propose a statistically principled alternative based on an exact Bayesian procedure in which priors are explicitly defined on the parameters of the underlying AR process. Compared to GA, the Bayesian approach enhances recognition accuracy at high SNRs, but is slightly less accurate at low SNRs.

Ecole Polytechnique Fédérale de Lausanne2008