NRT INGEST Software

The INGEST software called NRT (Near Real Time) can periodically download your data from remote devices and store it on the MCS (MOSAiC Central Storage). The software is installed on board RV Polarstern and has access to all networks on board and on the ice. Another instance is installed at AWI and collects e.g. remote sensing products from landside servers to the Land-MCS.

Recipes

So called recipes can be defined to gather your sensor's data, too. These can be scheduled to run e. g. every 5 minutes, every hour or every day at a certain time.

Protocols

Supported protocols to download data are:

Protocol	Description
FTP/FTPS/SFTP	The software will connect to these kind of remote servers e.g. your Linux-based embedded computer, scan given subfolders for new files and download them to a defined local folder. Patterns can be defined to only download files matching a certain expression. Examples: datamap_.jpg* scans for: datamap_0001.jpg, datamap_0002.jpg, datamap_0003.jpg .... mydata_%YEAR%%MONTH%%DAY%_.dat* scans for: mydata_20190806_235900.dat, mydata_20190806_235900.dat, ... and can store those files locally to an according daily folder /mcs/mysensor/%YEAR%/%MONTH%/%DAY%.
SMB	Connect to Windows SMB or Linux SAMBA shares and retrieve files from there. This works especially well for AWI-/VPN-internal servers/VMs where data is stored. The file patterns work, too, as described above in section FTP/FTPS/SFTP.
HTTP	The software will connect to a URL and download data e.g. from a web service or HTTP server. Date and time patterns such as %YEAR%, %MONTH% or %DAY% can be used in the URL to be able e.g. to gather daily extractions from a data web service. HTTP Basic Auth and URL-based authentification is supported.
IMAP	The software will connect to a mail account, will scan the inbox for emails having certain From or Subject patterns and download those and/or attachments to a defined local folder. The local folder definition may also contain date patterns in order to sort incoming files directly to daily or monthly sub folders: /mcs/mysensor/%YEAR%/%MONTH%/%DAY%.
RSYNC	The software will connect to a remote RSYNC server and mirror a given remote folder to a local folder e.g. on MCS.

Postprocessing routines

In every recipe you can define routines called script calls to be executed after each complete download of a file. These can be all Linux-based tools such as shell commands and scripts or Python scripts and (currently) need to be installed on the INGEST software's server.
Certain patterns such as %FILE% can be included in those script calls to allow passing arguments to your scripts. Example:

python3 /scriptpath/convertGpsRinexToLatLon.py %FILE% # script call defined
python3 /scriptpath/convertGpsRinexToLatLon.py /mcs/mysensor/2019/08/06/mydata_20190806_235900.dat # script call executed by INGEST software

Please consider that scripts running on the INGEST software's server should only do simple conversion or file operation jobs and should not do high performance calculations.

Connect to O2A's DASHBOARD and Data Web Service

The INGEST software already provides connectors to the O2A modules DASHBOARD and Data Web Service. Incoming data can be forwarded automatically to those components.
For that it is necessary to provide your raw data in the NRT Data Format or program a driver script that fulfills conversion.

How to automate your download?

In your SENSOR portal find (or add!) your sensor and find the INGEST tab in the sensor overview.
Fill the Contact Form for Data Ingest. Make sure to choose in the top: RAW data should be transfered continuously in near realtime.