ComputingComputer ScienceComputer SystemsData ManagementDistributed SystemsStorage SystemsData Storage SystemsDistributed StorageDistributed Storage SystemsFile StorageFile SystemsStorage ManagementDistributed File System ArchitectureFile System PerformanceOver-provisioningSmall File ManagementStorage OptimizationStorage Performance Optimization
Nowadays, Hadoop distributed file system as a distributed storage system, has a good effect on the storage of large files. However, there is a natural flaw in the storage of small files: storing a large number of small files will produce excessive metadata, resulting in namenode memory bottlenecks; frequent RPC communications will cause time consumption due to over-provisioning.
5
The Hadoop Distributed File System
Konstantin V. Shvachko, Hairong Kuang, Sanjay Radia et al. · 2010 · 4.8K citations
Implementing WebGIS on Hadoop: A case study of improving small file I/O performance on HDFS
Xuhui Liu, Jizhong Han, Yunqin Zhong et al. · 2009 · 179 citations