I have multiple segments of data that look similar to the one in the image:
In the signal there is some stability, then some period where the signal is decreasing, and then the signal stabilizes again. I am trying to find a way to identify the intervals where the data is generally decreasing (the area between the red lines in the image), but the noise in the data has made this difficult for me.
I have tried using variance and a comparison of a simple moving average and exponential weighted moving average to find the points at which the decrease begins and ends, but the noise is messing things up, so my main questions are for any recommendations for a good way to smooth out the noise or even better if there are any suggestions on how to identify those decreasing intervals.
Broadly, you have three stages
We need to characterise these mathematically, we can do that many ways, but one which is somewhat intuitive is using the rolling standard deviation
n
samplesn
samplesn
samplesWe can easily compute the rolling standard deviation with a loop.
First, some dummy data:
rng(0); % fix random seed for repeatable example
N = 1e3; % number of samples
x = linspace(0,1,N); % fixed rate x axis data
y = rand(1,N)*0.4 + (x>0.5)*0.5; % noisy data with a step change in middle
y = movmean( y, N*0.08 ); % smooth out the data a bit
Now compute the std. dev. over some window. The length of this window will need tuning for your data, depending on how abrupt your step changes are relative to the sample rate.
n = 50; % window size
sd = zeros(size(y));
for ii = (n/2):(N-n/2)
% std dev over 'n' samples, centred around the iith point
sd(ii) = std(y(ii-(n/2)+1:ii+(n/2)));
end
Using a simple threshold which will also need tuning to your data, we can identify the areas of high-change:
thrDetect = 0.05; % threshold to detect change from std. dev.
bDetect = sd > thrDetect; % Boolean detection array
Plotting, we can check this works:
figure(1); clf; subplot( 2, 1, 1 ); hold on;
plot( x, y, '.', 'displayname', 'data' );
plot( x, bDetect, 'k', 'displayname', 'std. dev. over threshold' );
legend('show', 'location', 'best'); grid on;
subplot( 2, 1, 2 ); hold on;
plot( x, sd, 'displayname', 'Rolling std. dev' );
yline( thrDetect, 'displayname', 'threshold' );
legend('show', 'location', 'best'); grid on;