Data quality control
This page helps track bad, incomplete, or otherwise excluded data from SNOTEL sites.
Procedure for screening data files
- Run testfiles.m
- Run testsensors.m
- Flag any sensors that look like the data being read is in error.
- Flag any sensor that appear to have a large level shift in the middle of a water year
- DO NOT flag missing data (gaps), or short timeseries, this will be taken care of by testfiles.m and nancheck.m`
Directories and files
Both the NRCSdata/allsensors_dailly/ and the
NRCSdata/soilsensors_hourly/ directories contain a
sitelist.txt file that is generated by the sitelistgenerator.sh shell script.
When changes are made to these directories (files added or removed) this
script should be re-run.
- baddata.txt is present in allsensors_daily/. It lists all datafiles, by siteID and year, that should be excluded from analysis. The actual exclusion is performed by
- In both data directories (above) there are subdirectories that hold incomplete, empty, or otherwise problematic files.`
Bad data tracker
\^ SiteID \^ Years \^ Problem \^ in baddata.txt? \^ Other \^ | 306 | 1998-2004 | Sensors at wrong depth, plus an I-2 header | Yes | Seems to get excluded by loadsnotel | | 2029 | 2000,2001 | | | WARNING - I edited these files, removing the last line of data, because it was effing up my scripts | | 1008 | 2001 | | | This file has an empty time column which effs everything up | | 802 | 2006 | | | Not filtering with sigma filter for some reason | | 896 | 2005-6 | | | "" | | 1015 | 2005 | | | "" | | 1053 | 2005 | | | "" | | 828 | 2006 | | | Excluded because it has values over 45 degrees C | | 742 | multiple | | | Sensors out of order | | 926 | multiple | | | "" | | 969 | multiple | | | "" | | 592 | multiple | | | Very high temps in soil | | 593 | multiple | | | "" | | 766 | multiple | | | "" |