云环境下应用感知的动态重复数据删除机制

贺秦禄 1; 边根庆 1; 邵必林 2; 贾雷刚 1

doi:10.7652/xjtuxb201810004

您当前的位置：

首页 >

文章列表页 >

云环境下应用感知的动态重复数据删除机制

更新时间：2025-07-09

- 云环境下应用感知的动态重复数据删除机制
- A Dynamic Deduplication Method with Application-Aware in Cloud Environment
- 西安交通大学学报 2018年52卷第10期页码：24-30
- 作者机构：
  
  西安建筑科技大学信息与控制工程学院,西安,710055
- 作者简介：
- 基金信息：
  
  : 2018-01-27。作者简介: 贺秦禄(1982-),男,博士,讲师;边根庆(通信作者),男,副教授,硕士生导师。基金项目: 国家自然科学基金资助项目(61672416);陕西省自然科学基础研究计划资助项目(2018JM6105);西安建筑科技大学人才基金资助项目(RC1707)
- DOI：10.7652/xjtuxb201810004
  中图分类号： TP393.1
- 网络首发：2018-10-10，
  
  纸质出版：2018
- 稿件说明：
移动端阅览
贺秦禄 1, 边根庆 1, 邵必林 2, 等. 云环境下应用感知的动态重复数据删除机制[J]. 西安交通大学学报, 2018,52(10):24-30.

A Dynamic Deduplication Method with Application-Aware in Cloud Environment[J]. 2018, 52(10): 24-30.
贺秦禄 1, 边根庆 1, 邵必林 2, 等. 云环境下应用感知的动态重复数据删除机制[J]. 西安交通大学学报, 2018,52(10):24-30. DOI： 10.7652/xjtuxb201810004.

A Dynamic Deduplication Method with Application-Aware in Cloud Environment[J]. 2018, 52(10): 24-30. DOI： 10.7652/xjtuxb201810004.

摘要

针对传统在线/离线重删对云存储系统中重删效率不高的问题

采用混合重复数据删除(Hy-Dedup)机制

通过融合在线和离线两种方式进行有效的数据重删。该方案在线重删阶段根据负载类型对指纹索引进行聚类分组

设置不同重删阈值来评估数据流的空间局部一致性

提高了缓存命中率; 离线重删阶段采用延迟敏感的方法

对在线阶段缓存没有命中的重复块进行精确重删。通过这种混合方式在保持系统的I/O性能和吞吐量的前提下

显著减少了写入云存储的重复数据量。实验结果表明

与iDedup机制相比

Hy-Dedup机制可将在线重删率提高35.9%

磁盘空间需求减少41.36%

并且能够在云存储系统中实现高准确率的重删

提升重删效率

节省存储空间。

Abstract

A hybrid deduplication method(Hy-Dedup)is adopted to solve the problem that the deduplication efficiency in the cloud storage system is not high for traditional mode online/offline deduplication

and the method performs effective data deduplication by combining online and offline modes. This method clusters fingerprint indices according the type of loads in online deduplication stage by adopting the fingerprint caching technology. The temporal local consistency of the duplicated data in data stream is estimated and the spatial local consistency is evaluated by setting different deduplication thresholds to reduce the disk fragments. The problem that the cache cannot be hit because lack of local consistency in the offline deduplication phase will be solved. The duplicated data is significantly reduced by this method while maintaining the I/O performance and the system throughput. Experimental results and a comparison with iDedup show that Hy-Dedup improves the online deduplication ratio by up to 35.9% and the disk capacity requirement reduces by 41.36%. It is concluded that the proposed method can achieve high-deciding deduplication in the cloud storage system

improve deduplication efficiency

and save storage space.

关键词

Keywords

references

王龙翔, 张兴军, 朱国峰, 等. 重复数据删除中的无向图遍历分组预测方法 [J]. 西安交通大学学报, 2013, 47(10): 51-56.

WANG Longxiang, ZHANG Xingjun, ZHU Guofeng, et al. A grouping prediction method based on undirected graph traversal in de-duplication system [J]. Journal of Xi'an Jiaotong University, 2013, 47(10): 51-56.

NISHA T R, ABIRAMI S, MANOHAR E. Experimental study on chunking algorithms of data deduplication system on large scale data [C]∥Proceedings of the International Conference on Soft Computing Systems, Advances in Intelligent Systems and Computing. Berlin, Germany: Springer, 2016: 91-98.

付印金, 肖侬, 刘芳, 等. 基于重复数据删除的虚拟桌面存储优化技术 [J]. 计算机研究与发展, 2012, 49(S1): 125-130.

FU Yinjin, XIAO Nong, LIU Fang, et al. Deduplication based storage optimization technique for virtual desktop [J]. Journal of Computer Research and Development, 2012, 49(S1): 125-130.

SUDHAKARAN S, TREESA M. A survey on data deduplication in large scale data [J]. International Journal of Computer Applications, 2017, 165(1): 1-4.

XIA W, JIANG H, FENG D, et al. Similarity and locality based indexing for high performance data deduplication [J]. IEEE Transactions on Computers, 2015 64(4): 1162-1176.

SRINIVASAN K, BISSON T, GOODSON G, et al. iDedup: latency-aware, inline data deduplication for primary storage [C]∥Proceedings of the USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2012: 24.

MAO Bo, JIANG Hong, WU Suzhen, et al. POD: performance oriented I/O deduplication for primary storage systems in the cloud [C]∥Proceedings of the 2014 IEEE International Parallel and Distributed Processing Symposium. Piscataway, NJ, USA: IEEE, 2014: 767-776.

KAISER J, SÜß T, NAGEL L, et al. Sorted deduplication: how to process thousands of backup streams [C]∥ Proceedings of the 2016 32th Symposium on Mass Storage Systems and Technologies. Piscataway, NJ, USA: IEEE, 2016: 07897082.

FU Yinjin, JIANG Hong, XIAO Nong, et al. AA-dedupe: an application-aware source deduplication approach for cloud backup services in the personal computing environment [C]∥Proceedings of the IEEE International Conference on Cluster Computing. Piscataway, NJ, USA: IEEE, 2011: 112-120.

LI Wenji, JEAN-BAPTISE G, RIVEROS J, et al. Cachededup: in-line deduplication for flash caching [C]∥ Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2016: 301-314.

PARK D, RAN Ziqi, NAM Y J, et al. A lookahead read cache: improving read performance for deduplication backup storage [J]. Journal of Computer Science and Technology, 2017, 32(1): 26-40.

VITTER J S. Random sampling with a reservoir [J]. ACM Transactions on Mathematical Software, 1985, 11(1): 37-57.

VALIANT G, VALIANT P. Estimating the unseen: improved estimators for entropy and other properties [J]. Advances in Neural Information Processing Systems, 2013, 64(6): 2157-2165.

LIU Chuanyi, LU Yingping, SHI Chunhui, et al. ADMAD: application-driven metadata aware de-duplication archival storage system [C]∥Proceedings of the 2008 5th IEEE International Workshop on Storage Network Architecture and Parallel. Piscataway, NJ, USA: IEEE, 2008: 29-35.

NG C H, LEE P P C. RevDedup: a reverse deduplication storage system optimized for reads to latest backups [C]∥Proceedings of the 4th Asia-Pacific Workshop on Systems. New York, USA, ACM, 2013: 1-7.

LI Wenji, JEAN-BAPTISE G, RIVEROS J, et al. Cachededup: in-line deduplication for flash caching [C]∥Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York USA: USENIX, 2016: 301-314.

MANDAL S, KUENNING G, OK D, et al. Using hints to improve inline block-layer deduplication [C]∥Proceedings of the 14th USENIX Conference on File and Storage Technologies. New York, USA: USENIX, 2016: 315-322.

FAN Ziqi, WU Fenggang, PARK D, et al. Hibachi: a cooperative hybrid cache with NVRAM and DRAM for storage arrays [C]∥Proceedings of the 2017 33th IEEE Symposium on Mass Storage Systems and Technologies. Piscataway, NJ, USA: IEEE, 2017: 1-12.

杨一晨,张国和,梁峰,等.一种基于可编程逻辑器件的卷积神经网络协处理器设计.2018,52(7):153-159.[doi:10. 7652/xjtuxb201807022]

褚伟波,王丽芳,蒋泽军,等.面向资源使用的TTL缓存服务计费机制与存储策略.2018,52(6):42-47.[doi:10.7652/xjtuxb201806007]

王鑫,高家明,梁煜,等.一种高效存储多级二维9/7离散小波变换结构.2018,52(4):111-116.[doi:10.7652/xjtuxb2018 04016]

李杨,王劲林,叶晓舟,等.面向嵌入式处理器的优化Montgomery模乘算法.2017,51(2):47-52.[doi:10.7652/xjtuxb 201702008]

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向分散式存储的云存储安全架构

支持隐私保护的云存储框架设计

采用机器学习的大规模并行程序I/O性能建模与预测方法

混合电解槽阵列协同的风光耦合制氢系统双层多目标优化

超临界水异常区域热力学特性研究