Using nighttime lights for economic inference needs cleaning of the data.
There are negative observations in the data as a consequence of NOAA's cleaning procedures. Many researchers replace them with 0. It can be done using the following:
datacube = replace!(x -> x < 0 ? 0 : x, datacube) # replace all values below 0 with 0
These values can be replaced with missing and interpolated.
datacube = replace!(x -> x < 0 ? missing : x, datacube) # replace all values below 0 with missing
NighttimeLights.mark_missing
— MethodFor each pixel, NOAA produces monthly estimates of radiance using the mean of radiance measured on days considered free of clouds. Radiance on months with no cloud-free observations are marked as zero, these should be marked as missing or or not available. The mark_missing function uses the radiance and the cloud-free observations image to marks missing wherever there no 0 cloud-free observations.
radiance = rand(1:10.0, 10, 10)
cloud = rand(0:5, 10, 10)
mark_missing(radiance, cloud)
NighttimeLights.mark_missing
— MethodThe mark_missing function also works on datacubes.
radiance = rand(1:10.0, 10, 10, 10)
cloud = rand(0:5, 10, 10, 10)
mark_missing(radiance, cloud)
Wherever the number of cloud-free observations is 0, radiance will be marked as missing.
NighttimeLights.linear_interpolation
— MethodUses linear interpolation to fill for missing values.
Example:
x = rand(1:10.0, 10)
x[5] = missing
linear_interpolation(x)
NighttimeLights.outlier_mask
— MethodThere are extremely high values in the data due to fires, gas flare etc. You may find some values even greater than the aggregate radiance of large cities. Such pixels also have high standard deviation. These pixels may not be of importantance from the point of view of measureming prosperity. The outlier_mask
function generates a mask of pixels with standard deviation less that the 99.9th percentile. Essentially, this function can be used to removed top 1 percent of pixels by standard deviation. A mask can be provided to the function, so that it calculates the percentile based on the lit pixel of the mask. For example, if the datacube is a box around India and the mask is the polygon mask of India, the outlier_mask function will calculate the 99th percentile of the standard deviation of the pixels inside India's boundary.
outlier_mask(datacube, mask)
NighttimeLights.outlier_ts
— FunctionThe time series of a pixel may show a few outliers, but as a whole the pixel may be of importantance in measuring economic activity. The outlier_ts function uses replaces the outlier observations with interpolated values. This is done using the tsclean function the forecast package of R.
sample_timeseries = datacube[1, 2, :] # The time series of pixel [1, 2]
outlier_ts(sample_timeseries)
NighttimeLights.background_noise_mask
— FunctionPixels with no economic activity may show some light due to background noise. These pixels could be in forests, oceans, deserts etc. The background_noise_mask
function generates a background moise mask such that those pixels which are considered dark are marked as 0 and those considered lit are marked as 1. The function uses the datacubes of radiance and clouds to generate annual image of the last year the data. The function considers all the pixels below a provided threshold as dark and remaining to be lit.
background_noise_mask(radiance_datacube, clouds_datacube)
NighttimeLights.bias_correction
— MethodClouds months tends to have lower radiance due to attenuation. The bias_correction function uses the number of cloud-free observations to adjust the radiance accordingly.
bias_correction(radiance, clouds)
NighttimeLights.bias_correction
— MethodThe bias correction function can use the datacubes of radiance and the number of cloud-free observations to correct for attenuation in radiance due to low number of cloud-free observations.
bias_correction(radiance, clouds)
NighttimeLights.conventional_cleaning
— MethodAll steps of data cleaning that most researchers do can be performed using the conventional cleaning funciton.
Example
radiance_datacube = rand(1:1000, 10, 10, 10)
clouds_datacube = rand(1:1000, 10, 10, 10)
conventional_cleaning(radiance_datacube, clouds_datacube)
NighttimeLights.PatnaikSTT2021
— MethodThe PatnaikSTT2021 function performs all the steps of the new cleaning procedure described in But clouds got in my way: Bias and bias correction of VIIRS nighttime lights data in the presence of clouds, Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas as conventional cleaning.
Example
radiance_datacube = rand(1:1000, 10, 10, 10)
clouds_datacube = rand(1:1000, 10, 10, 10)
PatnaikSTT2021(radiance_datacube, clouds_datacube)