JSON API response with pages

geoakkawi · June 27, 2020, 2:37pm

I’m querying a REST API which send back the data in pages, the API specifies the total number of pages in the headers.
I cannot figure out how to fetch all the pages and not a specific page

k4s1m · June 28, 2020, 1:34am

The Redash API uses paging. I’m not aware of a command that sends everything. The way redash_toolbelt gets around this is recursion:

github.com

getredash/redash-toolbelt/blob/fbf9e7116a430092a7b1bccf9a98681b4edddbd9/redash_toolbelt/client.py#L73-L89


def paginate(self, resource):
    """Load all items of a paginated resource"""
    stop_loading = False
    page = 1
    page_size = 100

    items = []

    while not stop_loading:
        response =  resource(page=page, page_size=page_size)

        items += response['results']
        page += 1

        stop_loading = response['page'] * response['page_size'] >= response['count']

    return items

geoakkawi · June 28, 2020, 5:33am

Thank you for the response, please excuse my ignorance, but I’m not sure if we are on the same page.
What I’m trying to achieve is through the JSON Datasource in Redash

I’m getting data from an API which supplies the data in pages, is there a way to loop through the pages? is there a way to write any kind of logic inside that?
This is the API I’m calling:

k4s1m · June 29, 2020, 1:03am

Sorry I misunderstood.

Redash doesn’t support what you’re talking about. I’d recommend making a proxy between Redash and the API that unfurls the data. Then point Redash at the proxy.

geoakkawi · June 29, 2020, 2:01am

Any suggestions on how to do that? Best practices? Easiest ways that I can look into? Tools?
Any suggestions would be great

▼Categories

▼Tags

JSON API response with pages