...
Supported protocols to download data are:
Protocol | Description |
---|---|
FTP/FTPS/SFTP | The software will connect to these kind of remote servers e.g. your Linux-based embedded computer, scan given subfolders for new files and download them to a defined local folder.
|
HTTP | The software will connect to a URL and download data e.g. from a web service or HTTP server. Date and time patterns such as %YEAR%, %MONTH% or %DAY% can be used in the URL to be able e.g. to gather daily extractions from a data web service. HTTP Basic Auth and URL-based authentification is supported. |
IMAP | The software will connect to a mail account, will scan the inbox for emails having certain From or Subject patterns and download those and/or attachments to a defined local folder. The local folder definition may also contain date patterns in order to sort incoming files directly to daily or monthly sub folders: /mcs/mysensor/%YEAR%/%MONTH%/%DAY%. |
RSYNC | The software will connect to a remote RSYNC server and mirror a given remote folder to a local folder e.g. on MCS. |
In every recipe you can define routines called script calls to be executed after each complete download of a file. These can be all Linux-based tools such as shell commands and scripts or Python scripts and (currently) need to be installed on the INGEST software's server.
Certain patterns such as %FILE% can be included in those script calls to allow passing arguments to your scripts. Example:
...
Please consider that scripts running on the INGEST software's server should only do simple conversion or file operation jobs and should not do high performance calculations.
The INGEST software already provides connectors to the O2A modules DASHBOARD and Data Web Service. Incoming data can be forwarded automatically to those components.
For that it is necessary to provide your raw data in the NRT Data Format or program a driver script that fulfills conversion.
...