Python Processor Error "no dataset generated", print log shows data

Processors, Python, Spark, R
1

Hey,

I am using a Single Input Python Processor to make several (~80) Calls to an API which gets me 100 results each call. Those are concated after parsing the json response with pandas.
Up to about 50 Calls it worked in all my tests. There is no clear/unique number which triggers this error (i.e. one test with 53 calls worked, another failed). The processor shows the following error:

type: "ProcessorExecutionErrorDTO"
message: "Python Script didn't generate any dataset on output. If that's intentional switch 'Generate Empty Dataset Output' toggle ON to generate an empty dataset on output."

There is no print output shown when this error occurs. When I toggle the option for “generate empty dataset”, I can see that the data appears to be fully/correctly loaded via a print statement at exactly the line where normally the output would be defined.
Anybody seen something like this or has an idea what could cause this?

2 Likes
2

I encountered the same issue, is there a solution to this?

3

In my case it was due to the data set size of the result table. Increasing the pydata memory limit to 4GB solved the issue

1 Like
4

I had the same problem, too, and I spent a long time wondering what might be the case. Fortunately, I found this thread here, and leant that memory limitations might cause this. And indeed, it turned out that this was the cause for my problem.
It really would be great to have an error message in place to inform the user - as it seems that many users were in similar situations.

2 Likes