Data quality control

This page helps track bad, incomplete, or otherwise excluded data from SNOTEL sites.

See also: Project overview, Programming documentation, Data collection/prep, Data analysis log

Procedure for screening data files

  • Run testfiles.m
  • Run testsensors.m
  • Flag any sensors that look like the data being read is in error.
  • Flag any sensor that appear to have a large level shift in the middle of a water year
  • DO NOT flag missing data (gaps), or short timeseries, this will be taken care of by testfiles.m and nancheck.m`

Directories and files

Both the NRCSdata/allsensors_dailly/ and the NRCSdata/soilsensors_hourly/ directories contain a sitelist.txt file that is generated by the sitelistgenerator.sh shell script. When changes are made to these directories (files added or removed) this script should be re-run.

  • baddata.txt is present in allsensors_daily/. It lists all datafiles, by siteID and year, that should be excluded from analysis. The actual exclusion is performed by loadsnotel.m (for now).
  • In both data directories (above) there are subdirectories that hold incomplete, empty, or otherwise problematic files.`

Bad data tracker

\^ SiteID \^ Years \^ Problem \^ in baddata.txt? \^ Other \^ | 306 | 1998-2004 | Sensors at wrong depth, plus an I-2 header | Yes | Seems to get excluded by loadsnotel | | 2029 | 2000,2001 | | | WARNING - I edited these files, removing the last line of data, because it was effing up my scripts | | 1008 | 2001 | | | This file has an empty time column which effs everything up | | 802 | 2006 | | | Not filtering with sigma filter for some reason | | 896 | 2005-6 | | | "" | | 1015 | 2005 | | | "" | | 1053 | 2005 | | | "" | | 828 | 2006 | | | Excluded because it has values over 45 degrees C | | 742 | multiple | | | Sensors out of order | | 926 | multiple | | | "" | | 969 | multiple | | | "" | | 592 | multiple | | | Very high temps in soil | | 593 | multiple | | | "" | | 766 | multiple | | | "" |