Connect OD to SFTP

1

image
image1303×409 8.37 KB

Goal (cf. picture): OD als ETL Tool:

  1. OD picks up data from SFTP
  2. OD does some ETL
  3. OD saves as .parquet on the cluster.
  4. OD provides the parquet impala-ready for microstrategy (tableau, PowerBI, …)

While 1-3 is possible, is it currently possible to connect OD to a SFTP, more concreteley:

I know from other customers, that we can use mounted volumes to use the Filesystem R/W Feature.

  1. Can we use that feature as well for SFTP?
  2. If not, can we mount an SFTP to OD such that we can use the feature?
  3. If not, is the feature planned?

If 1 to 3 are positive, how do I use it?

2

OD has no SFTP support by itself as far as I know.

If you have sufficient control over to the underlying linux system of the instance you can use SSHFS and mount your sftp share somewhere into the filesystem, then use fs r/w on it. I think it’s at least likely that it will work well (but needs to be tried out if it does of course).

Another option could be to use the python processor and some sftp library but definitely keep performance in mind here if expect larger amounts of data.

3

Thank you @peter.dahlberg