Unbiased estimation of standard deviation

In statistics and in particular statistical theory, unbiased estimation of a standard deviation is the calculation from a statistical sample of an estimated value of the standard deviation (a measure of statistical dispersion) of a population of values, in such a way that the expected value of the calculation equals the true value. Except in some important situations, outlined later, the task has little relevance to applications of statistics since its need is avoided by standard procedures, such as the use of significance tests and confidence intervals, or by using Bayesian analysis. However, for statistical theory, it provides an exemplar problem in the context of estimation theory which is both simple to state and for which results cannot be obtained in closed form. It also provides an example where imposing the requirement for unbiased estimation might be seen as just adding inconvenience, with no real benefit. Motivation In statistics, the standard deviation of a popul

SurvJamda: an R package to predict patients' survival and risk assessment using joint analysis of microarray gene expression data

Haleh Yasrebi

SurvJamda (Survival prediction by joint analysis of microarray data) is an R package that utilizes joint analysis of microarray gene expression data to predict patients' survival and risk assessment. Joint analysis can be performed by merging datasets or meta-analysis to increase the sample size and to improve survival prognosis. The prognosis performance derived from the combined datasets can be assessed to determine which feature selection approach, joint analysis method and bias estimation provide the most robust prognosis for a given set of datasets. AVAILABILITY: The survJamda package is available at the Comprehensive R Archive Network, http://cran.r-project.org. CONTACT: hyasrebi@yahoo.com.

Oxford University Press2011

Bridging the gap between model-driven and data-driven methods in the era of Big Data

Gael Lederrey

Data-driven and model-driven methodologies can be regarded as competitive fields since they tackle similar problems such as prediction. However, these two fields can learn from each other to improve themselves. Indeed, data-driven methodologies have been developed to use advanced methodologies based on Big Data technologies. On the other hand, model-driven methodologies concentrate on developing mathematical models based on theory and expert knowledge to allow for interpretability and control. Through three main contributions, this thesis aims to bridge the gap between these two fields by using their strengths and applying them to its counterpart.Discrete Choice Models (DCMs) have shown tremendous success in many fields, such as transportation. However, they have not evolved to tackle the growing amount of available data. On the other hand, Machine Learning (ML) researchers have developed optimization algorithms to efficiently estimate complex models on large datasets. Similarly, faster estimation of DCMs on larger datasets would improve the efficiency of modelers as well as enable new research axes. Thus, we take inspiration from the large body of existing research in efficient parameter estimation with extensive data and large numbers of parameters in deep learning and apply it to DCMs. The first chapter of this thesis introduces the HAMABS algorithm, which combines three fundamental principles to enable faster parameter estimation of DCMs (20x speedup compared to standard estimation) without compromising the precision of the parameter estimates.Collecting large amounts of data can be cumbersome and costly, even in the era of Big Data. For example, ML researchers in Computer Vision have been developing generative deep learning models to augment datasets. DCM researchers face similar issues with tabular data, e.g. travel surveys. In addition, if the collection process is not performed correctly, these datasets can contain bias, lack consistency, or be unrepresentative of the actual population. The second chapter of this thesis introduces the DATGAN, a Generative Adversarial Network (GAN) integrating expert knowledge to control the generation process. This new architecture allows modelers to generate controlled and representative synthetic data, outperforming similar state-of-the-art generative models. Finally, researchers are increasingly developing fully disaggregate agent-based simulation models, which use detailed synthetic populations to generate aggregate passenger flows. However, detailed disaggregate socioeconomic data is usually expensive to collect and heavily restricted in terms of access and usage. As such, synthetic populations are typically either drawn randomly from aggregate level control totals, limiting their quality, or tightly controlled, limiting their application and usefulness. To combat this, the third chapter extends the DATGAN methodology to generate highly detailed and consistent synthetic populations from small sample data. First, ciDATGAN learns to generate the variables in a low-sample highly detailed dataset, e.g. household travel survey. It then completes a high-sample dataset with few variables, e.g. microdata census, by generating the previously learned variables. The results show that this methodology can correct for bias and may enable the transfer of synthetic populations to new areas/contexts.

EPFL2022

Design and Experimental Validation of an LTE-Based Synchrophasor Network in a Medium Voltage Distribution Grid

Asja Derviskadic, Mario Paolone, Marco Pignati, Paolo Romano, Lorenzo Zanni

We present and experimentally validate in a real-scale medium voltage (MV) grid a synchrophasor network that exploits the availability of a public 4G LTE communication infrastructure. An 18 buses, 10 kV feeder located in Huissen, The Netherlands, has been equipped with 10 Phasor Measurement Units (PMUs) connected to the MV grid by means of dedicated voltage and current sensors. The PMUs stream synchrophasor data through a public 4G LTE network via an information-centric networking-based middleware, named C-DAX. The measurements are received and time-aligned at a phasor data concentrator and fed to a real-time state estimation application. The paper presents the various field-trial components and validates the feasibility of exploiting the 4G LTE technology for PMU-based applications. Specifically we assess the performance of the adopted wireless telecommunication infrastructure with and without the C-DAX middleware, as well as the accuracy of the real-time state estimation process.

Power Systems Computation Conference2018

Bridging the gap between model-driven and data-driven methods in the era of Big Data

Gael Lederrey

EPFL2022