Recommended storage locations
Overview
When you need more disk space you have four options:
- Option 1: Delete (unnecessary data)
- Option 2: netscratch (temporary data)
- Option 3: Tape Archive (cold data)
- Option 4: Isilon (online data)
Option 1: Delete (unnecessary data) | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
What is not needed anymore can be deleted with the Windows Explorer or | |||||||||||||||||||||
| Option 2: /netscratch (Temporary data) | |||||||||||||||||||||
Features: 50 TB space available for up to 90 days (automatically deleted afterwards) at
Documentation: FileSharingvianetscratch | |||||||||||||||||||||
| Option 3: Tape Archive (HSM, cold data) | |||||||||||||||||||||
Features: Cheap (nearly) unlimited space available, two data copies in building D and E for security reasons.
Note 1: In case of a disaster (e.g., building E burns down) it might take some weeks/months to recover all data. | |||||||||||||||||||||
Option 4: Isilon (online data) | |||||||||||||||||||||
Features: Fast central network storage, daily snapshots (kept for six month, protection against user errors), erasure coding (protection against hardware failure), replication partially to Potsdam/Bremerhaven or HSM (protection against facility disaster)
Note 2: The personal data (4a) is replicated to the Isilon at the other location (Bremerhaven/Potsdam). Note 3: The project data (4b) can be replicated to the HSM or not (your choice in https://cloud.awi.de/#/projects), so data can be restored in case of a disaster. Note 4: Data on the Isilon is accessible from (parts of) ollie and could be processed using HPC. | |||||||||||||||||||||
Option 5: Nextcloud ( Fileshareing with internal and external user ) | |||||||||||||||||||||
Features: Filesharing and collaborative working on office files (collabora online). User can share files with users that are external users without an account. Every user has 1GB of nextcloud storage quota and up to 100GB on SMB share (Option 4a). For the SMB shares nextcloud is like a bridge to the data. You can also get access via network drive on your window file browser. EXTERNAL STORAGE
Backup / Restore dataThe backup is handled by the ISILON itself. To restore files you can use on Windows filesystems the option "previous versions" to get older backups. You can also get your files online in the nextcloud environment. To do so you need to modify your URL. Connect ProjectsWhen you create a project and get storage from cloud.awi.de, you can access the file on your storage via nextcloud. Internal Storage
Backup / Restore dataThe data is backed up via VMWARE. A backup is created once a night. It is possible to restore individual files. This can take some time because the VMWARE administrator has to restore the files from the backup. |
The exact path to your project data is shown in https://cloud.awi.de/#/projects this table is just for a quick overview to understand the syntax:
| Online storage | (Linux-)Path | |||
|---|---|---|---|---|
| Location | Bremerhaven | /isibhv/projects | ||
| Potsdam | /isipd/projects | |||
| Access | Internal | |||
| External (only available for Bremerhaven) | -dmz | |||
| Replica | Tape | |||
| No replica | -noreplica | |||
| Tape storage/archive | ||||
| /hs/projects/ | ||||
Access:
The platform independent usage of the filesystems in Bremerhaven and Potsdam is possible by the protocols
- Network File System (NFS, Linux)
- Server-Message-Block-Protokoll (SMB , Windows & Mac)
- sftp (Linux, Windows, Mac, but tape storage only)
Linux
The NFS filesystems are provided by an automounter and mounted as soon as they are required. The user just has to know the filesystems (e.g., /isibhv/projects/), which he wants to access.
Windows
On Windows & Mac, the user has to connect manually to the network drives from the File Explorer/Finder using the SMB-Server:
- Right click on This PC in the Windows File Explorer
- Select Map network drive.
- Enter \\smb-server\share (e.g.: \\smb.isibhv.dmawi.de\home)
- Users connecting from a external network using VPN, will need to use the option Connect using different credentials. Your AWI-Username and password are needed, for example:
Name: dmawi\wwichtig or wwichtig@dmawi.de
Passwort: xxxxxx
- External connections are connections via the Internet, connections from our internal wireless networks awi-guest and eduroam or our special network for private notebooks.
Mac-Users
For a manual connection from a Mac you will need to use the
- menu option go to => connect with server and enter the following:
- smb://server/share (e.g.: smb://smb.isibhv.dmawi.de/home)
After that, a login prompt will appear. Please note, the field "Workgroup/Domain" has to contain the domain name dmawi.
Snapshots (Backups)
Linux/Unix-Users
In every directory of the file server nfs.isibhv.dmawi.de, there is a hidden subdirectory called .snapshot. Subdirectories, with the creation date in their name, contain the snapshots in question.
Windows-Users
For every directory of the file server smb.isibhv.dmawi.de, you can right click on the directory and access the snapshots via the menu previous versions. The previous versions are sorted by the creation date of that snapshot.
File Sharing via netscratch
Windows, Linux and MacOS manage and interpret access permissions to files in different ways, sometimes not compatible with each other. For this reason two different /netscratch directories are provided:
/netscratch | /netscratch/public \\smb.isibhv.dmawi.de\netscratch | |
|---|---|---|
| Information | Everything you store here is, at first, only changeable by you. | Everything you store here is, at fist, readable, changable and deletable by everyone |
Windows | New files and directorties are only readable by you. You can add further users or groups via the Windows-Fileexplorer, in the tab "Properties → Security", but this will compromise or overwrite the previously defined Linux permissions. | You can change the permissions by using the Windows-Fileexplorer, in the tab - "Properties → Security" |
Linux | New files and directores are readable by everyone. You can change the access permissions for the owner and group by using chmod and/or chgrp. | You can change the permissions with chmod and chgrp, but this will compromise or overwrite the previously defined Windows permissions. |
MacOS | See Windows | See Windows |
Quota
You can check the occupied disk space with
Windows: File Explorer -> right click -> Properties
Linux
du-sh <path># shows physical disk usage (including data protection, accounted for quota)
find <path> -ls | awk '{++c; s+=$7} END {print c " Files\n", s/1024^4 " TB"}' # Note: This necessary on the HSM system, where data might be offline.du-sh --apparent-size<path> # shows logical space- df -h <path> # shows quota and apparent size
- Projects: Project volume and quota can be checked with cloud.awi.de/#/projects