...
Recommended storage locations
Overview
When you need more disk space you have four options:
- Option 1: Delete (unnecessary data)
- Option 2: / netscratch (temporary data)
- Option 3: Tape Archive (cold data)
- Option 4: Isilon (online data)
Option 1: Delete (unnecessary data) | |||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
What is not needed anymore can be deleted with the Windows Explorer or | |||||||||||||||||||||||||||
Option 2: /netscratch (Temporary data) | |||||||||||||||||||||||||||
Features: 50 TB space available for up to 90 days (automatically deleted afterwards) at
Documentation: | https://intranet.awi.de/infrastruktur/rechenzentrum/dienste/file-service.html|||||||||||||||||||||||||||
Option 3: Tape Archive (HSM, cold data) | |||||||||||||||||||||||||||
Features: Cheap (nearly) unlimited space available, two data copies in building D and E for security reasons. | HSM, hssrv1
| ${FS}
|
Note 1: In case of a disaster (e.g., building E burns down) it might take some weeks/months to recover all data. | ||||||||||||||||||||||||
Option 4: Isilon (online data) | |||||||||||||||||||||||||||
Features: Fast central network storage, daily snapshots (kept for six month, protection against user errors), erasure coding (protection against hardware failure), replication partially to Potsdam/Bremerhaven or HSM (protection against facility disaster)Documentation: https://intranet.awi.de/infrastruktur/rechenzentrum/dienste/file-service.html
Note 2: The personal data (4a) is replicated to the Isilon at the other location (Bremerhaven/Potsdam). Note 3: The project data (4b) is can be replicated to the HSM or not (your choice in https://cloud.awi.de/#/projects), so data can be restored in case of a disaster. Note 4: Data on the Isilon is accessible from (parts of) ollie and could be processed using HPC. | |||||||||||||||||||||||||||
Option 5: Nextcloud ( Fileshareing with internal and external user ) | |||||||||||||||||||||||||||
Features: Filesharing and collaborative working on office files (collabora online). User can share files with users that are external users without an account. Every user has 1GB of nextcloud storage quota and up to 100GB on SMB share (Option 4a). For the SMB shares nextcloud is like a bridge to the data. You can also get access via network drive on your window file browser. EXTERNAL STORAGE
Backup / Restore dataThe backup is handled by the ISILON itself. To restore files you can use on Windows filesystems the option "previous versions" to get older backups. You can also get your files online in the nextcloud environment. To do so you need to modify your URL. Connect ProjectsWhen you create a project and get storage from cloud.awi.de, you can access the file on your storage via nextcloud. Internal Storage
Backup / Restore dataThe data is backed up via VMWARE. A backup is created once a night. It is possible to restore individual files. This can take some time because the VMWARE administrator has to restore the files from the backup. |
The exact path to your project data is shown in https://cloud.awi.de/#/projects this table is just for a quick overview to understand the syntax:
Online storage | (Linux-)Path | |||
---|---|---|---|---|
Location | Bremerhaven | /isibhv/projects | ||
Potsdam | /isipd/projects | |||
Access | Internal | |||
External (only available for Bremerhaven) | -dmz | |||
Replica | Tape | |||
No replica | -noreplica | |||
Tape storage/archive | ||||
/hs/${FS}projects/ ${FS} depends on your section |
Access:
The platform independent usage of the filesystems in Bremerhaven and Potsdam is possible by the protocols
- Network File System (NFS, Linux)
- Server-Message-Block-Protokoll (SMB , Windows & Mac)
- sftp (Linux, Windows, Mac, but tape storage only)
Linux
The NFS filesystems are provided by an automounter and mounted as soon as they are required. The user just has to know the filesystems (e.g., /isibhv/projects/), which he wants to access.
...
In every directory of the file server nfs.isibhv.dmawi.de, there is a hidden subdirectory called .snapshot. Subdirectories, with the creation date in their name, contain the snapshots in question.
Windows
...
-Users
For every directory of the file server smb.isibhv.dmawi.de, you can right click on the directory and access the snapshots via the menu previous versions. The previous versions are sorted by the creation date of that snapshot.
...
Windows: File Explorer -> right click -> Properties
Linux
du
-sh
<path>
# shows physical disk usage (including data protection, accounted for quota)
find <path> -ls | awk '{++c; s+=$7} END {print c " Files\n", s/1024^4 " TB"}' # Note: This necessary on the HSM system, where data might be offline.
(accounted for quota)du
-sh --apparent-size
<path> # shows logical space- df -h <path> # shows quota and apparent size
- Projects: Project volume and quota can be checked with cloud.awi.de/#/projects
...