Versions Compared

Old Version 14

changes.mady.by.user Malte Thoma

Saved on

compared with

New Version Current

changes.mady.by.user Michael Günster

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

Recommended storage locations

Overview

When you need more disk space you have four options:

  • Option 1: Delete (unnecessary data)
  • Option 2: / netscratch (temporary data)
  • Option 3: Tape Archive (cold data)
  • Option 4: Isilon (online data)
 https://intranet.awi.de/infrastruktur/rechenzentrum/dienste/file-service.htmlHSM, hssrv1Cost ${FS}
${FS} is one of {bsys|csys|gsys|tech}50 Note 2: There will be no charge for relocated data (from unixfs/winfs) until further notice (right of continuance, "Bestandsschutz")150

Option 1: Delete (unnecessary data)

What is not needed anymore can be deleted with the Windows Explorer or rm -rf /<YOUR_DIRECTORY>

Option 2: /netscratch (Temporary data)
Features: 50 TB space available for up to 90 days (automatically deleted afterwards) at 
  • smb:\\smb.isibhv.dmawi.de\netscratch\public (Windows/MacOS/Linux) known as public in Windows
  • /isibhv/netscratch (Linux)

Documentation:

FileSharingvianetscratch

Option 3: Tape Archive (HSM, cold data)

Features: Cheap (nearly) unlimited space available, two data copies in building D and E for security reasons.
Documentation:  HSM (hssrv1): Archiving (and storing) Data (

hsm.dmawi.de)

OptionUtilisationPathNoteRemovedDefault Quota
Budget (per TB and yr)
3aProject datahssrv1.awi.de:/hs/
projects/project_name

You can create a project and request HSM-resources here: https://cloud.awi.de/#/projects
Short Documentation: https://spaces.awi.de/x/Zz5aFQ

Three months after  project ends
-25 €
3bPermanent storagehssrv1.awi.de:/hs/usero/Submit data to PANGAEA via https://www.pangaea.de/submit/never∞internal accounting

Note 1:  In case of a disaster (e.g., building E burns down) it might take some weeks/months to recover all data.

Option 4: Isilon (online data)

Features: Fast central network storage, daily snapshots (kept for six month, protection against user errors), erasure coding (protection against hardware failure), replication partially to Potsdam/Bremerhaven or HSM (protection against facility disaster)Documentation: https://intranet.awi.de/infrastruktur/rechenzentrum/dienste/file-service.html

Option

Utilisation

Path

Note

Removed

Default

Quota

Cost

Budget (per TB and yr)

4aPersonal data

UNIX: /home/
UNIX:smb:\\smb.isibhv.dmawi.de/personal
WIN:smb:\\smb.isibhv.dmawi.de/home
WIN:smb:\\smb.isibhv.dmawi.de/personal

The separation /home vs. /personal is partly historic, based on differences beween Windows and Unix.
Three months
90 days after user left AWI100 GB (fix)free
4bProject data

UNIX: /isibhv/

${PP}

projects[-noreplica]/project_name
WIN:smb:\\smb.isibhv.dmawi.de\

${PP}

projects[-noreplica]/project_name

${PP} depends on the location (Bhv|Pot), access (internal|external) and replica (none|tape|other-location). See table below.

You can create a project and request
Online
online/Isilon resources here: https://cloud.awi.de/#/projects
Three months after project ends

Short Documentation: https://spaces.awi.de/x/Zz5aFQ		90 days after project expires-

100-125 €

Note 1: This data "isibhv" indicates the Isilon in Bremerhaven. For Potsdam this would be "isipd".
Note 2: The personal data (4a) is replicated to the Isilon in Potsdam, which ensures availability in case of a disaster in Bremerhavenat the other location (Bremerhaven/Potsdam).
Note 2: Option 3 (Tape Archive) could be used in addition to Option 3 for additional data protection 3: The project data (4b) can be replicated to the HSM or not (your choice in https://cloud.awi.de/#/projects), so data can be restored in case of a disaster.
Note 3: There will be no charge for relocated data (from unixfs/winfs) until further notice (right of continuance, "Bestandsschutz")
Note 4: Data on the Isilon is accassible accessible from (parts of) ollie and could be processed using HPC.

Option 5: Nextcloud ( Fileshareing with internal and external user )

Features: Filesharing and collaborative working on office files (collabora online). User can share files with users that are external users without an account. Every user has 1GB of nextcloud storage quota and up to 100GB on SMB share (Option 4a). For the SMB shares nextcloud is like a bridge to the data. You can also get access via network drive on your window file browser.

EXTERNAL STORAGE

Shares Mountable directories(Projects must be mouted by an administrator)
smb://smb.isibhv.dmawi.deprojects, personal/<username>, projects-dmz
smb://smb.isipd.dmawi.de

Backup / Restore data

The backup is handled by the ISILON itself. To restore files you can use on Windows filesystems the option "previous versions" to get older backups. You can also get your files online in the nextcloud environment. To do so you need to modify your URL.

Connect Projects

When you create a project and get storage from cloud.awi.de, you can access the file on your storage via nextcloud.

Internal Storage

/var/www/nextcloud/dataQuota of 1GB per user

Backup / Restore data

The data is backed up via VMWARE. A backup is created once a night. It is possible to restore individual files. This can take some time because the VMWARE administrator has to restore the files from the backup.

The exact path to your project data is shown in https://cloud.awi.de/#/projects this table is just for a quick overview to understand the syntax:

Potsdam/Bremerhaven
Online storage
(Linux-)Path
LocationBremerhaven/isibhv/projects


Potsdam/isipd/projects

AccessInternal



External
(only available for Bremerhaven)
-dmz
ReplicaTape



-hsmNo replica

-noreplica
Tape storage/archive



/hs/projects/


Access:

The platform independent usage of the filesystems in Bremerhaven and Potsdam is possible by the protocols

  • Network File System (NFS, Linux)
  • Server-Message-Block-Protokoll (SMB , Windows & Mac)
  • sftp (Linux, Windows, Mac, but tape storage only)

Linux

The NFS filesystems are provided by an automounter and mounted as soon as they are required. The user just has to know the filesystems (e.g., /isibhv/projects/), which he wants to access.

Windows

On Windows & Mac, the user has to connect manually to the network drives from the File Explorer/Finder using the SMB-Server:

  • Right click on This PC in the Windows File Explorer
  • Select Map network drive.
  • Enter  \\smb-server\share  (e.g.: \\smb.isibhv.dmawi.de\home)


Info
  • Users connecting from a external network using VPN, will need to use the option Connect using different credentials. Your AWI-Username and password are needed, for example:
    Name:       dmawi\wwichtig   or   wwichtig@dmawi.de
    Passwort: xxxxxx
  • External connections are connections via the Internet, connections from our internal wireless networks awi-guest and eduroam or our special network for private notebooks.

Mac-Users

For a manual connection from a Mac you will need to use the

  • menu option go to => connect with server and enter the following:
  • smb://server/share (e.g.: smb://smb.isibhv.dmawi.de/home)

After that, a login prompt will appear. Please note, the field "Workgroup/Domain" has to contain the domain name dmawi.

Snapshots (Backups)

Linux/Unix-Users

In every directory of the file server nfs.isibhv.dmawi.de, there is a hidden subdirectory called .snapshot. Subdirectories, with the creation date in their name, contain the snapshots in question.

Windows-Users

For every directory of the file server smb.isibhv.dmawi.de, you can right click on the directory and access the snapshots via the menu previous versions. The previous versions are sorted by the creation date of that snapshot.

File Sharing via netscratch

Windows, Linux and MacOS manage and interpret access permissions to files in different ways, sometimes not compatible with each other. For this reason two different /netscratch directories are provided:


/netscratch
recommended for  Linux

/netscratch/public
recommended for Windows 

\\smb.isibhv.dmawi.de\netscratch

Information

Everything you store here is, at first, only changeable by you.

Everything you store here is, at fist, readable, changable and deletable by everyone

Windows

New files and directorties are only readable by you. You can add further users or groups via the Windows-Fileexplorer, in the tab "Properties → Security", but this will compromise or overwrite the previously defined Linux permissions.

You can change the permissions by using the Windows-Fileexplorer, in the tab - "Properties → Security"

Linux

New files and directores are readable by everyone. You can change the access permissions for the owner and group by using chmod and/or chgrp.

You can change the permissions with chmod and chgrp, but this will compromise or overwrite the previously defined Windows permissions.

MacOS

See Windows

See Windows

Quota

You can check the occupied disk space with

  • Windows: File Explorer -> right click -> Properties

  • Linux

    • du -sh                 <path> # shows physical disk usage (including data protection, accounted for quota)
      find <path> -ls | awk '{++c; s+=$7} END {print c " Files\n", s/1024^4 " TB"}' # Note: This necessary on the HSM system, where data might be offline.

    • du -sh --apparent-size <path> # shows logical space

    • df -h                                      <path>  # shows quota and apparent size
    • Projects: Project volume and quota can be checked with cloud.awi.de/#/projects

Related articles

Content by Label
showLabelsfalse
max5
showSpacefalse
sortmodified
reversetrue
cqllabel = "kb-how-to-article" and type = "page" and space = "HELP"

...