An optimization strategy of massive small files storage based on HDFS

Xun Cai, Cai Chen, Yi Liang

2018 · 14 citations · 5 references

DOIFull text

Open access

Concepts

Abstract

Nowadays, Hadoop distributed file system as a distributed storage system, has a good effect on the storage of large files. However, there is a natural flaw in the storage of small files: storing a large number of small files will produce excessive metadata, resulting in namenode memory bottlenecks; frequent RPC communications will cause time consumption due to over-provisioning.

References

5