Pagination in get_content method Python SDK

Processors, Python, Spark, R Python
1

Dear fellow Python-SDK Users,
I was wondering if there is a (tested and sustainable) solution for the following problem I have

When using od_api.datatables.get_content() the pagination argument has a default of 1000. If I want to load the entire dataset (and I don’t know the exact row count before hand…), is there any simple solution to do so?

Thanks in advance :slight_smile:

2

Hey @melchior.rb :slight_smile:

You should be able to edit the pagination_size by handing over that parameter to the function like this:

content: DataTableContent = onedata_api.datatables.get_content(
                data_id=dt["datatable_id"],
                pagination_size=MAX_NUMBER_OF_ROWS_TO_FETCH_FROM_OD
            )

I think you cannot keep this flexible so I usually set it to a size that would for sure fit the dataset (but you of course run into problems if the data is too big because you are trying to fetch it all in 1 request).

3

Hey @magdalena.murr :slight_smile:

Thanks for the answer. Just for me to understand: How do you set MAX_NUMBER_OF_ROWS_TO_FETCH_FROM_OD do you get it by doing something like this:

MAX_NUMBER_OF_ROWS_TO_FETCH_FROM_OD = onedata_api.datatables.get_content(
                data_id=dt["datatable_id"],
                pagination_size=1
            ).total_count

and then use the variable in your code to define the pagination_size?

DataTableContent = onedata_api.datatables.get_content(
                data_id=dt["datatable_id"],
                pagination_size=MAX_NUMBER_OF_ROWS_TO_FETCH_FROM_OD
            )

Or do you just set it to e.g. 10000000000?

4

Sorry for the late reply!

In our case we simply set it something really big but if it makes sense you could of course also do it the way you described :slight_smile: The only problem is that the row-count metadata is not available for all datasets necessarily (VDTs for example do not provide this as metadata).