Where can I store/archive large amounts of data?

Question

I am supervising a research project in which several terabytes of data are to be generated and analysed. Within the project, a suitable data recording and data storage concept is currently being developed. Is it possible for you to host storage space of several terabytes for a research project? If this is possible: What are the requirements for this?

Answer 1

NFDI itself does not provide any hardware that can be used to store data. The first stop would probably be to check whether your university/research institution provides adequate storage via their in-house repository. 

We also provide the "data collections explorer", a database that can be used to find a fitting repository. It currently includes roughly 90 entries, both national and international.

There's also the option to set-up your own solution. Today, several terabytes fit quite comfortably into a RAID system, which some departments have purchased for their purposes. This also guarantees data security. Local systems are often much better suited to analysing the data than pure repositories which are mainly designed to make it easy to find published data (e.g. Coscine). Such systems are neither intended nor suitable for active use for data analysis. If you have further questions, I can connect you to our matter experts in the consortium.

Answer 2

Not a direct answer to the questions, but the researchers behind the NFDI4Ing archetype DORIS [https://nfdi4ing.de/archetypes/doris/] might have the same challenge when working with data in High-Performance Computing (HPC). Maybe they are be a good starting point for contact.

And as a side comment: Don't forget to plan a respective financial budget in your research proposal to storage and archive such an amount of data.

Answer 3

If you have a NFDI project or you are part of the universities of NRW, then you can apply for storage at Coscine (briefly mentioned by @TSchwetje). This allows you get up to 125 TB (or more) of an object storage system. Please keep in mind that Coscine **is not a repository** and therefor not suitable for publishing data (in contradiction to the other answer here).

Thanks to the interface of Coscine you will be able to share the data with colleagues to continously work on it.

The requirements are: The project is a scientific/research project and you have to fill out an application form (link goes directly to the S3 storage, there is also a general overview of the differen resource types) to get the storage granted.

The whole process is described on the webpage, or in the article:

Lang, Ilona et al.: Carrots and Sticks: Motivating with Storage for Good RDM – Science Led Allocation of Research Data Storage Resources within an Integrated RDM System, in Heuveline, Vincent, Bisheh, Nina und Kling, Philipp (Hrsg.): E-Science-Tage 2023: Empower Your Research – Preserve Your Data, Heidelberg: heiBOOKS, 2023, S. 140–148.  (doi.org/10.11588/heibooks.1288.c18071)