I'm kind of starting in Javascript and I need help to figure out how can I make this code synchronous while looping through for loop. Basically what I'm doing is making multiple POST requests inside for loops and then im scrapping the data using the library X-Ray and finally I'm saving the result to a Mongo Database. The output is ok but it comes in unordered way and suddenly hangs and I have to force close using ctrl+C. This is my function:
function getdata() {
const startYear = 1996;
const currentYear = 1998; // new Date().getFullYear()
for (let i = startYear; i <= currentYear; i++) {
for (let j = 1; j <= 12; j++) {
if (i === startYear) {
j = 12;
}
// Form to be sent
const form = {
year: `${i}`,
month: `${j}`,
day: '01',
};
const formData = querystring.stringify(form);
const contentLength = formData.length;
// Make HTTP Request
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded',
},
uri: 'https://www.ipma.pt/pt/geofisica/sismologia/',
body: formData,
method: 'POST',
}, (err, res, html) => {
if (!err && res.statusCode === 200) {
// Scrapping data with X-Ray
x(html, '#divID0 > table > tr', {
date: '.block90w',
lat: 'td:nth-child(2)',
lon: 'td:nth-child(3)',
prof: 'td:nth-child(4)',
mag: 'td:nth-child(5)',
local: 'td:nth-child(6)',
degree: 'td:nth-child(7)',
})((error, obj) => {
const result = {
date: obj.date,
lat: obj.lat.replace(',', '.'),
lon: obj.lon.replace(',', '.'),
prof: obj.prof == '-' ? null : obj.prof.replace(',', '.'),
mag: obj.mag.replace(',', '.'),
local: obj.local,
degree: obj.degree,
};
// console.log(result);
upsertEarthquake(result); // save to DB
});
}
});
}
}
}
I guess I have to use promises or callbacks but I can't understand how to do this, and I already tried using async await but with no success. If any additional info needs to be provided please tell me, thanks.
You are calling the request inside a loop.
Async functions are functions where getting the result (A.K.A., receiving a response in the callback function) is made after the main thread logic has ended.
This way, if we have this:
for (var i = 0; i < 12; i++) {
request({
data: i
}, function(error, data) {
// This is the request result, inside a callback function
});
}
The logic will be to run over the 12 request
s before calling the callbacks, so the callbacks will be stacked and called after all the main loop is run.
Without entering all the ES6 generators thing (As I think that it makes it a bit more complicated and also learning at low-level what is happening is better for you), what you have to do is to call the request
, wait for his callback function to be called and call the next request
. How to do that? There's a bunch of ways, but I usually go like this:
var i= 0;
function callNext() {
if (i>= 12) {
requestEnded();
} else {
request({
data: i++ // Increment the counter as we are not inside a for loop that increments it
}, function(error, data) {
// Do something with the data, and also check if an error was received and act accordingly, which is very much possible when talking about internet requests
console.log(error, data);
// Call the next request inside the callback, so we are sure that the next request is ran just after this request has ended
callNext();
})
}
}
callNext();
requestEnded() {
console.log("Yay");
}
Here you see the logic. You have a function named callNext
which will make the next call or call requestEnded
if no more calls are needed.
When request
is called inside callNext
, it will wait for the callback to be received (which will happen asynchronously, sometime in the future), will process the data received and then inside the callback you tell him to call again callNext
.