Multi-step automatic process

Briefly, the multi-step process was designed as follow:

1. Step 1 - Valid structure: selection of the time series with valid data structure

2. Step 2 - Length in years: selection of the time series with more than 105 recorded years

3. Step 3 - Recorded months: selection of the time series with more than 1280 recorded months (non missing)

4. Step 4 - Missing months: selection of the time series with less than 80 missing months

5. Step 5 - Length in months: selection of the time series with full monthly length not inferior to 1290 months

Step 1 - Selection by valid structure

Using the information table previusly built (T0_Information), we selected all the time series with the variable Status equal to OK.

Here the results of the selection procedure for valid structure (step 1):

##  Cleaning data - Step : 1 
##  ************************************ 
##  1035 accepted time series over 1260 
##  Current step acceptance rate : 82.14 % 
##  Loss from step 1: 17.86 %

Step 2 - Selection by length (years)

The second step of our selection procedure was to identify all the time series that had more than 106 recorded years.

The results of the second step of the selection procedure follows here:

##  Cleaning data - Step : 2 
##  ************************************ 
##  376 accepted time series over 1035 
##  Current step acceptance rate : 36.33 % 
##  Loss from step 1: 70.16 %

At this point, our subset consists of 376 weather stations.

Information table 2 - Monthly information

In order to apply step 3-5 of the selection procedure, we need to gather information about the data at a monthly level.

We then built a second information table, with the following fields:

• Country and Station name for each time series

• First_Year and First_Year, first and the last year of the recorded time series

• Years, length in years of the time series

• Months, number of available months (non missing)

• Full_Length, number of months between the first recorded month and the last one.

• Missing_M, number of missing months

As an example, consider the the first 10 rows of the T02_Information2 table:

Step 3 - Selection by available months

We can now refine the selection procedure checking that the number of non missing months is superior to 1260.

Here the results of this selection step:

##  Cleaning data - Step : 3 
##  ************************************ 
##  329 accepted time series over 376 
##  Current step acceptance rate : 87.5 % 
##  Loss from step 1: 73.89 %

Step 4 - Selection by missing months

We can now exclude all the time series with number of missing months above 80.

Recall that the previous step was about non missing months: in this step we are avoiding time series that (although long), contain so many “holes” that they might compromise the data quality.

Here the results of this selection step:

##  Cleaning data - Step : 3 
##  ************************************ 
##  278 accepted time series over 329 
##  Current step acceptance rate : 84.5 % 
##  Loss from step 1: 77.94 %

Step 5 - Length in months

The last step of the selection procedure is to check if the total length of the time series (from the first recorded month to the last one, missing included) is not inferior to 1290.

Above the results:

##  Cleaning data - Step : 5 
##  ************************************ 
##  277 accepted time series over 278 
##  Current step acceptance rate : 99.64 % 
##  Loss from step 1: 78.02 %

At this point we have a subset of 277 waether stations, characterized by a valid data structure and a length and presence of missing observation below specific thresholds.

Final data

Summary statistics of the selected of 96 time series is shown in Appendix B, table B1.

There are two time series that did not pass the tests but we still included them in our analysis. One is from Ivittuut, Greenland, and the second one is from Uppsala, Sweden. The reason is that Greenland is of particular interest in the climate debate, and the temperature series from Uppsala is the longest time series ever recorded.

All the selected time series are available in the data/final folder. All the names (country and stations) have been translated to English.

We have highlighted results from 9 weather stations because they are well known cities with good quality of the temperature data.