A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems

Nathanaël Cheriere (1, 2) , Matthieu Dorier (3) , Gabriel Antoniu (2)

1 ENS Rennes - École normale supérieure - Rennes
2 KerData - Scalable Storage for Clouds and Beyond
3 ANL - Argonne National Laboratory [Lemont]

Efficient resource utilization becomes a major concern as large-scale computing infrastructures such as supercomputers and clouds keep growing in size. Malleability, the possibility for resource managers to dynamically increase or decrease the amount of resources allocated to a job, is a promising way to save energy and cost. However, state-of-the-art parallel and distributed storage systems have not been designed with malleability in mind. The reason is mainly the supposedly high cost of data transfers required by resizing operations. Nevertheless, as network and storage technologies evolve, old assumptions about potential bottlenecks can be revisited. In this study, we model the duration of the commission operation, for which we obtain theoretical lower bounds. We then consider HDFS as a use case, and we show that our lower bound can be used to evaluate the performance of the commission algorithms. We show that the commission in HDFS can be greatly accelerated. With the highlights provided by our lower bound, we suggest improvements to speed the commission in HDFS.

Mots clés: Elastic Storage; Distributed File System; Malleable File System; Lower Bound; Commission

Domaines: Calcul parallèle, distribué et partagé [cs.DC]

Nathanaël Cheriere, Matthieu Dorier, Gabriel Antoniu. A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems. [Research Report] Irisa. 2018. ⟨hal-01817638v1⟩