Pervasive Data Analytics for Sustainable Energy Systems

With an ever growing population, global energy demand is predicted to keep increasing. Furthermore, the integration of renewable energy sources into the electricity grid (to reduce carbon emission and humanity's dependency on fossil fuels), complicates efforts to balance supply and demand, since their generation is intermittent and unpredictable. Traditionally, it has always been the supply side that has adapted to follow energy demand, however, in order to have a sustainable energy system for the future, the demand side will have to be better managed to match the available energy supply. In the first part of this thesis, we focus on understanding customers' energy consumption behavior (demand analytics). While previously, information about customer's energy consumption could be obtained only with coarse granularity (e.g., monthly or bimonthly), nowadays, using advanced metering infrastructure (or smart meters), utility companies are able to retrieve it in near real-time. By leveraging smart meter data, we then develop a versatile customer segmentation framework, track cluster changes over time, and identify key characteristics that define a cluster. Additionally, although household-level consumption is hard to predict, it can be used to improve aggregate-level forecasting by first segmenting the households into several clusters, forecasting the energy consumption of each cluster, and then aggregating those forecasts. The improvements provided by this strategy depend not only on the number of clusters, but also on the size of the customer base. Furthermore, we develop an approach to model the uncertainty of future demand. In contrast to previous work that used computationally expensive methods, such as simulation, bootstrapping, or ensemble, we construct prediction intervals directly using the time-varying conditional mean and variance of future demand. While analytics on customer energy data are indeed essential to understanding customer behavior, they could also lead to breaches of privacy, with all the attendant risks. The first part of this thesis closes by exploring symbolic representations of smart meter data which still allow learning algorithms to be performed on top of them, thus providing a trade-off between accurate analytics and the protection of customer privacy. In the second part of this thesis, we focus on mechanisms for incentivizing changes in customers' energy usage in order to maintain (electricity) grid stability, i.e., Demand Response (DR). We complement previous work in this area (which typically targeted large, industrial customers) by studying the application of DR to residential customers. We first study the influence of DR baselines, i.e., estimates of what customers would have consumed in the absence of a DR event. While the literature to date has focused on baseline accuracy and bias, we go beyond these concepts by explaining how a baseline affects customer participation in a DR event, and how it affects both the cust