A Dynamic Deduplication Method with Application-Aware in Cloud Environment

doi:10.7652/xjtuxb201810004

您当前的位置：

首页 >

文章列表页 >

A Dynamic Deduplication Method with Application-Aware in Cloud Environment

更新时间：2025-07-09

- A Dynamic Deduplication Method with Application-Aware in Cloud Environment
- Vol. 52, Issue 10, Pages: 24-30(2018)
- 作者机构：
  
  西安建筑科技大学信息与控制工程学院,西安,710055
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb201810004
  CLC： TP393.1
- Online First：10 October 2018，
  
  Published：2018
- 稿件说明：
移动端阅览
A Dynamic Deduplication Method with Application-Aware in Cloud Environment[J]. 2018, 52(10): 24-30.
DOI：

A Dynamic Deduplication Method with Application-Aware in Cloud Environment[J]. 2018, 52(10): 24-30. DOI： 10.7652/xjtuxb201810004.

摘要

针对传统在线/离线重删对云存储系统中重删效率不高的问题

采用混合重复数据删除(Hy-Dedup)机制

通过融合在线和离线两种方式进行有效的数据重删。该方案在线重删阶段根据负载类型对指纹索引进行聚类分组

设置不同重删阈值来评估数据流的空间局部一致性

提高了缓存命中率; 离线重删阶段采用延迟敏感的方法

对在线阶段缓存没有命中的重复块进行精确重删。通过这种混合方式在保持系统的I/O性能和吞吐量的前提下

显著减少了写入云存储的重复数据量。实验结果表明

与iDedup机制相比

Hy-Dedup机制可将在线重删率提高35.9%

磁盘空间需求减少41.36%

并且能够在云存储系统中实现高准确率的重删

提升重删效率

节省存储空间。

Abstract

A hybrid deduplication method(Hy-Dedup)is adopted to solve the problem that the deduplication efficiency in the cloud storage system is not high for traditional mode online/offline deduplication

and the method performs effective data deduplication by combining online and offline modes. This method clusters fingerprint indices according the type of loads in online deduplication stage by adopting the fingerprint caching technology. The temporal local consistency of the duplicated data in data stream is estimated and the spatial local consistency is evaluated by setting different deduplication thresholds to reduce the disk fragments. The problem that the cache cannot be hit because lack of local consistency in the offline deduplication phase will be solved. The duplicated data is significantly reduced by this method while maintaining the I/O performance and the system throughput. Experimental results and a comparison with iDedup show that Hy-Dedup improves the online deduplication ratio by up to 35.9% and the disk capacity requirement reduces by 41.36%. It is concluded that the proposed method can achieve high-deciding deduplication in the cloud storage system

improve deduplication efficiency

and save storage space.

关键词

Keywords

references

王龙翔, 张兴军, 朱国峰, 等. 重复数据删除中的无向图遍历分组预测方法 [J]. 西安交通大学学报, 2013, 47(10): 51-56.

WANG Longxiang, ZHANG Xingjun, ZHU Guofeng, et al. A grouping prediction method based on undirected graph traversal in de-duplication system [J]. Journal of Xi'an Jiaotong University, 2013, 47(10): 51-56.

NISHA T R, ABIRAMI S, MANOHAR E. Experimental study on chunking algorithms of data deduplication system on large scale data [C]∥Proceedings of the International Conference on Soft Computing Systems, Advances in Intelligent Systems and Computing. Berlin, Germany: Springer, 2016: 91-98.

付印金, 肖侬, 刘芳, 等. 基于重复数据删除的虚拟桌面存储优化技术 [J]. 计算机研究与发展, 2012, 49(S1): 125-130.

FU Yinjin, XIAO Nong, LIU Fang, et al. Deduplication based storage optimization technique for virtual desktop [J]. Journal of Computer Research and Development, 2012, 49(S1): 125-130.

SUDHAKARAN S, TREESA M. A survey on data deduplication in large scale data [J]. International Journal of Computer Applications, 2017, 165(1): 1-4.

XIA W, JIANG H, FENG D, et al. Similarity and locality based indexing for high performance data deduplication [J]. IEEE Transactions on Computers, 2015 64(4): 1162-1176.

SRINIVASAN K, BISSON T, GOODSON G, et al. iDedup: latency-aware, inline data deduplication for primary storage [C]∥Proceedings of the USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2012: 24.

MAO Bo, JIANG Hong, WU Suzhen, et al. POD: performance oriented I/O deduplication for primary storage systems in the cloud [C]∥Proceedings of the 2014 IEEE International Parallel and Distributed Processing Symposium. Piscataway, NJ, USA: IEEE, 2014: 767-776.

KAISER J, SÜß T, NAGEL L, et al. Sorted deduplication: how to process thousands of backup streams [C]∥ Proceedings of the 2016 32th Symposium on Mass Storage Systems and Technologies. Piscataway, NJ, USA: IEEE, 2016: 07897082.

FU Yinjin, JIANG Hong, XIAO Nong, et al. AA-dedupe: an application-aware source deduplication approach for cloud backup services in the personal computing environment [C]∥Proceedings of the IEEE International Conference on Cluster Computing. Piscataway, NJ, USA: IEEE, 2011: 112-120.

LI Wenji, JEAN-BAPTISE G, RIVEROS J, et al. Cachededup: in-line deduplication for flash caching [C]∥ Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2016: 301-314.

PARK D, RAN Ziqi, NAM Y J, et al. A lookahead read cache: improving read performance for deduplication backup storage [J]. Journal of Computer Science and Technology, 2017, 32(1): 26-40.

VITTER J S. Random sampling with a reservoir [J]. ACM Transactions on Mathematical Software, 1985, 11(1): 37-57.

VALIANT G, VALIANT P. Estimating the unseen: improved estimators for entropy and other properties [J]. Advances in Neural Information Processing Systems, 2013, 64(6): 2157-2165.

LIU Chuanyi, LU Yingping, SHI Chunhui, et al. ADMAD: application-driven metadata aware de-duplication archival storage system [C]∥Proceedings of the 2008 5th IEEE International Workshop on Storage Network Architecture and Parallel. Piscataway, NJ, USA: IEEE, 2008: 29-35.

NG C H, LEE P P C. RevDedup: a reverse deduplication storage system optimized for reads to latest backups [C]∥Proceedings of the 4th Asia-Pacific Workshop on Systems. New York, USA, ACM, 2013: 1-7.

LI Wenji, JEAN-BAPTISE G, RIVEROS J, et al. Cachededup: in-line deduplication for flash caching [C]∥Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York USA: USENIX, 2016: 301-314.

MANDAL S, KUENNING G, OK D, et al. Using hints to improve inline block-layer deduplication [C]∥Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2016: 315-322.

FAN Ziqi, WU Fenggang, PARK D, et al. Hibachi: a cooperative hybrid cache with NVRAM and DRAM for storage arrays [C]∥Proceedings of the 2017 33th IEEE Symposium on Mass Storage Systems and Technologies. Piscataway, NJ, USA: IEEE, 2017: 1-12.

杨一晨,张国和,梁峰,等.一种基于可编程逻辑器件的卷积神经网络协处理器设计.2018,52(7):153-159.[doi:10. 7652/xjtuxb201807022]

褚伟波,王丽芳,蒋泽军,等.面向资源使用的TTL缓存服务计费机制与存储策略.2018,52(6):42-47.[doi:10.7652/xjtuxb201806007]

王鑫,高家明,梁煜,等.一种高效存储多级二维9/7离散小波变换结构.2018,52(4):111-116.[doi:10.7652/xjtuxb2018 04016]

李杨,王劲林,叶晓舟,等.面向嵌入式处理器的优化Montgomery模乘算法.2017,51(2):47-52.[doi:10.7652/xjtuxb 201702008]

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Security Structure of Cloud Storage Based on Dispersal

Design of Cloud Storage Framework with Privacy-Preserving

Electrothermal Coupling Research on the Evolutionary Law of Anti-Corona Performance for High-Altitude Air-Cooled Generators

Integrated and Joint Design Optimization of Centrifugal Compressor Impeller and Casing Treatment at Multiple Working Points

Mechanism and Optimal Control Strategy for Hysteresis in Rapid Dehumidification Response of Indirect Thermal Systems in Electric Vehicles

Related Author

边根庆 1

高松 1

邵必林 2

黄汝维 1

桂小林 1

余思 1

张进 2

Related Institution

Department of Electronics and Information Engineering, Xi'an Jiaotong University

School of Computer, Electronics and Information

⁰