Scraping paged APIs

[kayakr]

Some endpoints only return a subset of the data for each request. Is it possibly to specify a URL pattern for paged data, or supply a json path or xpath to a pager element in the response, such that Flat could fetch (say) 32 pages with 100 rows in each page. Of course, next time the data is fetched, the dataset may have grown to 33 pages...

Also, how are errors, timeouts etc. reported?

[LoneRifle]

I currently work around this by using Netlify Functions (these are available on Netlify's free-tier) to handle the scraping of paginated data into a single JSON blob, and getting the Flat action to call that. More information can be found at @datascapesg.

[idan]

Hiya! There's no clean solution for this in Flat right now. You can abuse postprocessing to do it, but that's not a great solution.

Errors (including timeouts) are reported out as action run failures (run will have a red ❌). You can open up these runs and inspect their logs to see whatever exception was logged out as the run died.

[idan]

(If you'd like to contribute a paging mechanism to HTTP fetching in Flat, happy to shepherd that through to merging!)