expresspuppeteerpuppeteer-cluster

ProtocolError: Protocol error (Runtime.callFunctionOn): Target closed. - express.js and puppeteer-cluster


I want to iterate over key-pairs of data.extractRules and get elements data from the page.

This snippet inside forEach loop is causing app crashes. I tried hardcoding key and cssSelector and tried this outside of forEach loop and it worked.

    const extractContent = {};
    if (data.extractRules !== null) {
      Object.entries(data.extractRules).forEach(async ([key, cssSelector]) => {
        extractContent[key] = await page.$$eval(cssSelector, (elements) =>
          elements.map((element) => element.outerHTML)
        );
      });
    }

Solution

  • I figured out solution 😁

    
    async function getSelectorContent(page, cssSelector) {
      return page.$$eval(cssSelector, (elements) =>
        elements.map((element) => element.outerHTML)
      );
    }
    
    const extractContent = {};
    if (data.extractRules !== null) {
        await Object.entries(data.extractRules).reduce(
            async (item, [key, cssSelector]) => {
                await item;
                extractContent[key] = await getSelectorContent(page, cssSelector);
            },
            Promise.resolve()
        );
    }