The R Project - The Use of R in Official Statistics - uRos2019

Julie Josse Member of RFoundation and Rforwards, Professor of Statistics at Ecole Polytechnique, France.

Her first employment was in the statistics department of an Agronomy University (Agrocampus Ouest) where she was trained to « the French data analysis school » and had the opportunity to work closely with researchers from other departments and increases her interest in transversal studies. In the meantime, she prepared her PhD which was rewarded by the French Statistical Society as the best PhD in applied statistics. She has specialized in missing data, visualization and the nonparametric analyses of complex data structures. Her work was rewarded by a European Union grant in 2013 to increase her research potential and to spend a year at Stanford University. She spent a year as a researcher in INRIA before joining Polytechnique in 2016. At Polytechnique, she is actually responsible of a master data-sciences for business in collaboration with HEC. She has published over 30 articles and written 2 books in applied statistics. Her experience on dealing with incomplete data is recognized by the community: she organized the MissData conference on missing value in 2015 and she is often invited to give lectures to share her experience. Her vocation is to push methodological innovation to bring useful application of her research to the user in particular in bio-sciences and health. Julie Josse is dedicated to reproducible research with the R statistical software: she has developed packages including FactoMineR, denoiseR, missMDA to transfer her work, she is a member of the R foundation and of Rforwards to increase the participation of minorities in the community.

Abstract: A missing value tour in R

In many application settings, the data have missing features which make data analysis challenging. An abundant literature addresses missing data as well as more than 150 R packages. Funded by the R consortium, we have created the R-miss-tastic platform along with a dedicated task view which aims at giving an overview of main references, contributors, tutorials to offer users keys to analyse their data. This platform highlights that this is an active field of work and that as usual different problems requires designing dedicated methods.
In this presentation, I will share my experience on the topic. I will start by the inferential framework, where the aim is to estimate at best the parameters and their variance in the presence of missing data. Last multiple imputation methods have focused on taking into account the heterogeneity of the data (multi-sources with variables of different natures, etc.). Then I will present recent results in a supervised-learning setting. A striking one is that the widely-used method of imputing with the mean prior to learning can be consistent. That such a simple approach can be relevant may have important consequences in practice.

Giulio Barcaroli From 1977 to 2018 he has been working at the Italian National Institute of Statistics (Istat).

Until 1992 he was in the ICT Department. In 1992 he switched to the methodological sector, where he has been responsible for research and development of methods and techniques in the statistical production processes (in particular: computer assisted data collection, automated and assisted coding, editing and imputation of errors and missing data, data integration, sample design and estimation). In addition, he has been responsible for development activities of generalized software tools.
With respect to sample design he is one of the authors of the methodology for stratified sampling design implemented in the R package “SamplingStrata” (of which he is also the maintainer).
During this period he has held a series of positions: head of divisions (the last one: “Methods, Quality and Metadata”), member of Istat Methodologies Committee and of EUROSTAT DIME (Directors of Methodology in the European Statistical System Group).
He actively participated in the “modernization project” of Istat (he is one of the authors of the Istat’s Business Architecture) and also in the ESS standardization activities (three different ESSnet projects).
He has also been actively involved in the field of the use of Big Data sources for statistical purposes. In particular, he has been member of the Istat’s Technical Committee for the Statistical Use of Big Data, and he participated in the European Project “Essnet on Big Data”.
He is author of numerous publications, available at the following links:
https://www.researchgate.net/profile/Giulio_Barcaroli
https://scholar.google.it/citations?user=8G3UuoQAAAAJ&hl=en

Abstract: R at the Italian National Institute of Statistics (Istat): a twelve years story

It was in 2006-07 that a small group of people inside Istat began to consider the adoption of R as a valid alternative to the use of proprietary software like SAS. There were many reasons for that: R was free and open, and was on the research edge, while SAS was expensive and without the same research background. But the odds were not negligible: SAS was deep inside the production processes, hundreds of people had been trained and used SAS in a systematic way, the assistance levels seemed not to be comparable. Notwithstanding this, also because of the general movement towards open source software, Istat decided to at least diminish its dependence on proprietary solutions (in particular SAS-based). Two initiatives were launched: migration of generalized software systems using open technologies (and R was on the top of these), and mass training on R. Some adhered enthusiastically, some other opposed strong resistance. Nowadays R and SAS coexist, with some important advancements: a relevant number of researchers make use of R in daily activities, the great majority of generalized solutions are R-based, and an important project aiming at building a generalized architecture for statistical production will adopt services many of which are functions of R libraries. The overall balance can be said to be positive, and to report this experience, including concrete examples of how it evolved over time, can be useful to other OS Institutes that undergo or could undergo a similar transition.

Mark van der Loo and Edwin de Jonge, Statistics Netherlands
Statistical data cleaning for official statistics with R

Abstract: This tutorial will demonstrate how data quality can be systematically defined and improved using R. It will focus on data validation (data checking), locating errors, and imputing missing or erroneous values under restrictions. Examples will be drawn from the Structural Business Survey (SBS) where common restrictions include nonnegativity rules and record-wise balance checks. I will present short introduction to the main principles, provide quizzes and discussions for the audience, and give short R-based exercises. A number of R packages related to data validation will be demonstrated, including 'validate' (for data quality checks), 'errorlocate' (for error localization), 'simputation' (for imputation methods), 'rspa' (for value adjustment), and 'lumberjack' (for keeping track of changes in data). Special attention will be paid on how to combine the various data processing steps, and how to analyse and visualize the results. At the end of the course, participants will have insight into some of the methods common in data editing for business surveys as well as an overview of how to implement that with free and open source R and the mentioned packages.
https://github.com/data-cleaning/uRos2019_tutorial

References: MPJ van der Loo, E. de Jonge (2018) Statistical Data Cleaning with applications in R. John Wiley & Sons.

Alexander Kowarik and Bernhard Meindl, Statistics Austria
Modern package development in R

Abstract: In this tutorial, the aim is to learn about several tools that help you to improve building an maintaining R packages. Participants will learn about the following topics:
- setup continuous integration using travis
- perform automatic code and style checking
- improve unit tests in your package
- create better documentation
- automatically create nice looking documentation websites.
This tutorial will be "hands-on" so it is advised to bring your noteooks with you.
https://github.com/bernhard-da/uros19_tutorial

Marco Ballin and Giulio Barcaroli, Italian National Institute of Statistics (Istat)
Use of R package SamplingStrata for the Optimal Stratification of Sampling Frames for Multipurpose Sampling Surveys

Abstract: The aim of this tutorial is to enable the participants to learn how to use the R package “SamplingStrata” in order to optimize the design of stratified samples. The package offers an approach for the determination of the best stratification of a sampling frame, the one that ensures the minimum sample cost under the condition to satisfy precision constraints in a multivariate and multi-domain case. This approach is based on the use of the genetic algorithm: each solution (i.e. a particular partition in strata of the sampling frame) is considered as an individual in a population; the fitness of all individuals is evaluated applying the Bethel algorithm to calculate the sampling size satisfying precision constraints on the target estimates. Functions in the package allows to: (a) prepare necessary inputs and check their validity; (b) perform the optimization step choosing the values of the most important parameters; (c) assign the optimized strata labels to the sampling frame; (d) select a sample from the new frame accordingly to the best allocation; (e) test the compliance of the design to precision constraints.
https://github.com/barcaroli/SamplingStrata-Tutorial-uRos2019

Marcello D’Orazio, Italian National Institute of Statistics (Istat),
Integration of data sources in R through statistical matching

Abstract: Statistical matching (aka data fusion) consists in a set of techniques developed to integrate distinct data sources referred to the same target population (typically stemming out from sample surveys) to get insights on the relationship between variables not jointly observed in a single data source. The tutorial will give an idea of the standard setting of statistical matching and the corresponding underlying assumptions; it will show how to perform matching in R by means of facilities provided by the R package StatMatch, including analyses that should precede and follow the matching step.
https://github.com/marcellodo/StatMatch/tree/master/2019-05_Tutorial_uRos2019

Mervyn Ó'Lúing, Central Statistics Office, Ireland,
Mapping a Table of Data with Esri Shapefiles in R

Abstract: This is an introductory tutorial to mapping opensource data with two shapefiles, plot multiple time series maps and produce web based maps using the "ggplot2" and "tmap" R packages in R. Users will learn how to:
- create a CSV file with GDP data extracted from an official Tanzanian publication,
- create an R data-frame from this data,
- add a geospatial reference to this data-frame,
- download two Esri shapefiles, regions and water bodies, from the Tanzanian National Bureau of Statistics’ (TNBS) website,
- create geospatial data-frames from the Esri shapefiles,
- join the region geospatial data-frame to the GDP data-frame,
- plot the data using the “ggplot2” package, and overlay the region and water bodies geospatial data frames,
- plot multiple time series maps and produce interactive zoomable web based maps using the “tmap” package.