DEDUPLICATION WITH KEYS AND CHUNKS IN HDFS STORAGE PROVIDERS

computer science and engineering

Deduplication characterizes the elimination of duplicate or redundant information and it removes the repetitive information before storing it. These techniques are widely employed for data backup, network minimization, and storage overhead. Long established deduplication schemes have restrictions on encrypted data and security. In the proposed system, new deduplication techniques are employed efficiently. Instead of possessing periphrasis of a sole content, deduplication banishes redundant data by keeping only one physical copy and broaching other redundant data to that copy. Each one can be expounded based on nonidentical granularities which may be either way a whole file or a data block. MD5 and 3DES algorithm strengthen the technique. The methodology proposed here is (POF) of the file. Deduplication can now, properly address the reliability and tag consistency problem in HDFS storage systems. The proposed system succeeded in reducing the cost and time of uploading and downloading with storage space.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.