Split Definitions

The Scoopi definition file tends to grow as you scrape multiple types of data or large set of links. This chapter explains how to split the definitions into multiple files for easy maintenance.

Scoopi allows you to split the definitions either on job types or on top level elements.

Split on Job Types

The examples so far scraped BS, PL, snapshot data - price and snapshot - of Acme. We can split the definitions into three job types snapshot, bs and pl. The Example 13 splits job.yml into snapshot.yml, bs.yml and pl.yml. When we split on job types, the broad outline of each definition file is as below

locatorGroups:

  snapshotGroup:                -- locator group
    locators:                      -- locators
    ...

taskGroups:
  snapshotGroup:                -- task group
    priceTask:                     -- tasks
    ...

dataDefs:                       -- datadefs
  links:                           -- datadef

  ...

The files bs.yml and pl.yml do not specify locatorGroups as locators are created from linkGroup tasks in snapshot.yml.

Split on Top elements

Another way to split is based top elements - locatorGroups, taskGroups and dataDefs. The Example 14 splits top level elements into three files - locators.yml, tasks.yml and datadefs.yml.

When we split definitions, Scoopi merges all yml files found in the folder and creates effective defination.

In the next chapter we explain the logging framework of Scoopi.