By myself aka ingest.awi.de

Especially when you would like to submit one-time only data (e.g. missing data from a campaign, faulty re-processed data, etc.) we offer a way to do so without further stress.  If you would like to submit data for archival only go ahead without further restrictions. If you would also like to have your data available via dashboard.awi.de/data/, currently only data submission in NRT data format is supported. 

Go to ingest.awi.de, go to "Go to submission" (log in using tour sensor.awi.de credentials and you are offered a file structure (a tree of folders) that is  derived from the items where you are allowed to write to (as in you are the editor!). Search for the folder of your item. Go to data -> Year -> Month -> Day (if available) and drag and drop your data there. If no folders of Year, Month, Day is available you are requested to give i) an event label (e.g. PS130/2_35-1 -- took place 2022-05-24T00:49:11 so that ../data/2022/05/24 would be created), or ii) a date range (see figure) when you just drop files or a folder to the file structure. 

After pressing "Upload files", guess what, the files are uploaded. Furthermore a support ticket is generated so that you can keep track of the progress of your data.

Near real-time aka steady data stream with automated data ingest 

By clicking "Near real-time" the following dialogue offers the protocol of choice. 

All field names have an infotext available that appears when the mouse is hovered over. Please use it. ;)

No matter which protocol is used in the further steps there must be a file available to be tested! That means

• (at least) one mail must have been sent meeting the specified requirements.
• (at least) one file is placed at a SMB share.
• (at least) one file must be stored on a FTP server.
• (at least) one file is available via HTTP access. 

Whatever protocol is chosen, the last two entries are always the same. 

Interval: either minutes or hours specified by an integer. 
Dashboard Driver: choose one of the following

A little course on File naming Pattern

We offer some easy to use placeholder for date expressions (see table) which also have implications for the estimate age of the files (because the file check is run for the last 30 days solely). 

Furthermore any number from 0 to 9 can be met by [0-9] with a quantifier in curly braces, e.g. {3}. The same applies to characters (and as any operating to be taken seriously), while there is a difference between uppercase and lowercase. [A-Z] is any uppercase string, [a-z] stands for any lowercase string, and [a-zA-Z] implies a sequence of lower- or uppercase letters. Quantifiers apply here too.

Thus a three digit number is hit by [0-9]{3}. {,3} would be three or less (<100), {3,} three or more (>=100), [0-9]{2,3} would be any number between ten (10) and 999.
In addition, [a-z]{4} is a four letter lowercase string.

Asterisks (`*`) are not allowed.

For more background on regex have a look on pages as regex101.com/, cs.lmu.edu/~ray/notes/regex/, or tldp.org/LDP/abs/html/x17129.html.

Example: a file is named SVL_20221013_103003_ADCP-68636.nrt, hence 

• `_` is the separator,
• `SVL` indicates the station, 
• `20221013` represents the eight digits date string,
• `103003` is the six digits time string,
• `ADCP-68636` is the name of the item including serial number,
• `.nrt` is the suffix spaces.awi.de/x/OBOVEg

We could meet the name representation by several regex

Various protocols

Mail

Sender address: Who is the sender of this email.

Subject contains: The subject or parts of the subject to match

Save attachments: is the data contained in the attachment, if so, check box, otherwise leave blank (the preferred way)

Save mail body as text: is the data part of the mail body, if so, check box, otherwise leave blank (predominantly applies to buoys)

SMB

Server address: When the data is located on the isilon, the server address is `smb.isibhv.dmawi.de`, otherwise use the address if your dedicated server.
Port: Standard port is 445. Use a different one only if applicable.
Username: This is the name of the account with access to the share. Please avoid personal accounts and use functional accounts instead.
Password: The correspondig password to the above mentioned account.
Shared folder name: This is the name of the share itself.
Remote directory: This is the name of the folder that contains data to be ingested. It usually starts with "/".
File naming Pattern: see above

FTP

Server address: normal ftp address, standard AWI FTP is ftp.awi.de
Protocol: Choose FTP, SFTP, or FTPS, while SFTP is the preferred choice. 
Port: Use the port matching to the protocol. 
Username: This is the name of the account with access to the ftp. Please avoid personal accounts and use functional accounts instead.
Password: The correspondig password to the above mentioned account.
Remote directory: The path to the data starting with "/"

HTTP

tbc

Finalize your ingest

You can click "continue" when all tests to the right-hand side are passed. That includes

• connection can be established (SMB/FTP/HTTP),
• data is available with the given naming pattern,
• data can be downloaded,
• the file(s) contains data that can be read by the chosen driver.

If so (when all tests passed and the file head looks as expected), the user is shown the path where on isilon the data is stored (hint: the `%URNPATH%` is the complete urn of your item with the colons `:` substituted by slashes `/`). Additionally the ingested parameters can be mapped to meet the parameters defined in sensor.awi.de (see figure). Remark: Only the parameter long names are shown. Unused parameters are skipped for dashboard downloading. 

If everything is set and done the request needs to be submitted. If done, a support ticket at AWI helpdesk is generated i) to get the data managers moving, and ii) to keep track of the process for you. 

If you just push "Done" the recipe is stored (for later usage), but cannot not yet be used. It needs to be approved before using.