WhiteCollar

This page lists all the typical use cases encountered during whiteCollar development, not directly related to Marfeel's product.

• How to manage article order in a section with the whiteCollar
• How to extract lazy loaded content in a section with the whiteCollar
• How to manage section items leading to non-marfeelized content

Extract lazy loaded content from a section

When a tenant uses lazy loading strategies, some of the content might not be extracted by default in Marfeel.

To solve it, Puppeteer has a userInteraction library integrated that allows simulating a user interacting with the page before extracting its content.

Learn how to use the scrollpage function to extract lazy loaded content.

PhantomJS

When the tenant uses PhantomJS and can't be migrated to Puppeteer, a customization in the WhiteCollar is necesary to extract the lazy loaded content. Learn more about it in the dedicated guide.

Section items leading to non-marfeelized content

Use isExtractable: false

{
    selector: '.contenido .pestana:nth-of-type(2)',
    extractors: {
        title: R.always(' '),
        uri: R.always('https://example.com/mundial/historias'),
        isExtractable: false,
    },
    modifiers: []
}

Or delegating the decision to a custom function:

function isExtractable(node) {
    var uri = WC.qs('.link', node);
    var href = uri && uri.href;

    if (href && href.indexOf('sunday-times') < 0) {
        return false;
    }

    return true;
}
...
isExtractable: isExtractable
...