A framework for intra- and inter-group data management: the DIADEM system

Session

DHA.1 - Deep learning for analysis and interpretation of microscopy imaging data

Authors

Robert S. Pennington (2, 3, 1), Christoph T. Koch (1)

Affiliations

1. Department of Physics, Humboldt-Universität zu Berlin
2. Institute of Applied Physics, University of Tübingen, Auf der Morgenstelle 10
3. NMI Natural and Medical Sciences Institute at the University of Tübingen, Markwiesenstr. 55,

Keywords

Data storage, data transfer, metadata

Abstract text

For electron microscopy groups, data management is a necessary competency whose importance increases alongside improvements in high-data-rate hardware, complex analysis software, and multi-group collaborative efforts.However, not only are standards for data archiving lacking, but migrating data from one location to another often requires a patchwork of approaches that do not necessarily preserve chain-of-custody data integrity. This is a particular problem for electron microscopy: the electron microscope requires a sample, likely created by another group; different microscopes have different capabilities that can be combined on the same sample only if the experiments can easily span multiple machines and sites; and different microscope users have different procedures and assessments for data quality.

Thus, we have been developing the DIADEM system, described in this work. DIADEM stands for "Decentralized Integrated Archives of Data and Extended Metadata" (in English; "Dezentrale integrierte Archive von Daten und erweiterten Metadaten" in German). The DIADEM system consists of two main components: an easy-to-use secure local archive (the "DIADEM node") and the ability to easily transfer authorized data from a user at one site to a user at another site (the "DIADEM network"). The DIADEM node, by following a consistent set of policies, provides verifiable and redundant local storage with minimal overhead and maximal user-friendliness. The DIADEM network, by using a combination of novel software and proven systems, enables easy user-to-user data transfer both within one DIADEM node and between connected nodes (specifiable as either hub-and-spoke or direct-transfer). Additionally, the DIADEM design provides clear data integrity protocols covering both file corruption and data transfer information.

To summarize, the DIADEM system has been designed for the specific data usage and overhead needs of electron microscopists, while remaining secure through the use of specific policies. It includes the DIADEM node, for secure local data storage, and the DIADEM network, for inter-node secure data transfer.