采用资源划分的云环境下Hadoop资源许可调度方法

周墨颂 1; 朱正东 1; 董小社 1; 陈衡 1; 王寅峰 2

doi:10.7652/xjtuxb201508012

您当前的位置：

首页 >

文章列表页 >

采用资源划分的云环境下Hadoop资源许可调度方法

更新时间：2025-07-09

- 采用资源划分的云环境下Hadoop资源许可调度方法
- A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation
- 西安交通大学学报 2015年49卷第8期页码：69-74+108
- 作者机构：
  
  西安交通大学电子与信息工程学院,西安,710049
- 作者简介：
- 基金信息：
  
  : 2015-03-03。作者简介: 周墨颂(1988—),男,博士生;朱正东(通信作者),男,高级工程师,硕士生导师。基金项目: 国家自然科学基金资助项目(61173039);国家自然科学基金青年基金资助项目(61202041);国家高技术研究发展计划资助项目(2012AA01A306);深圳市科技计划资助项目(JCYJ20120615101127404)
- DOI：10.7652/xjtuxb201508012
  中图分类号： TP391
- 网络首发：2015-08-10，
  
  纸质出版：2015
- 稿件说明：
移动端阅览
周墨颂 1, 朱正东 1, 董小社 1, 等. 采用资源划分的云环境下Hadoop资源许可调度方法[J]. 西安交通大学学报, 2015,49(8):69-74+108.

A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation[J]. 2015, 49(8): 69-74+108.
周墨颂 1, 朱正东 1, 董小社 1, 等. 采用资源划分的云环境下Hadoop资源许可调度方法[J]. 西安交通大学学报, 2015,49(8):69-74+108. DOI： 10.7652/xjtuxb201508012.

A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation[J]. 2015, 49(8): 69-74+108. DOI： 10.7652/xjtuxb201508012.

摘要

针对云计算环境中Hadoop平台由于节点计算能力差异、多样混合负载共存等原因而出现的性能不佳的问题

提出一种采用资源划分的资源许可方法。该方法在云计算环境下通过减少资源浪费或负载过重等情况的出现来提高系统性能。该方法采集资源信息并推测任务资源需求

根据可用计算资源和任务需求动态划分、调度资源; 使用与资源无耦合的资源许可启动任务并控制任务数量调节资源利用率以适应云环境。使用该方法对比公平调度器在国家高性能计算中心(西安)进行测试发现:单作业在资源竞争环境中优于公平调度器的静态结果; 混合负载在3种测试环境中完成时间分别平均减少了27.5%、37.1%和50.98%

性能显著提升。实验结果表明

该方法可以适应负载资源需求和可用计算资源的变化

灵活划分计算资源

解决Hadoop在云环境中的性能不佳问题。

Abstract

A resource license scheduling method using resource allocation is proposed to improve the poor performance caused by different computing capacities of nodes and mixed workload in cloud computing. The method improves the performance through reducing the resources wasting or overload. It collects resource information and estimates resource requirements of workloads and allocates computing resources dynamically according to available resources and resource requirements of workloads. Licenses uncoupled with resources are used to launch tasks and to adjust the number of parallel tasks to adapt the cloud environment by controlling the number of licenses. The method is evaluated in the national high performance computing center(Xi'an). Results show that the completion times of single job workloads of the proposed method are better than those of the FAIR scheduler in competitive environments. Moreover

the completion times of mixed workloads of the method in three environments reduce 27.5%

37.1% and 50.98% respectively on average

that is

the performance of the method has a significant improvement. It can be concluded from the results that the proposed method adapts the complex environment and solve the performance problem in cloud computing.

关键词

Keywords

references

REISS C, TUMANOV A, GANGER G R, et al. Heterogeneity and dynamicity of clouds at scale: google trace analysis [C]∥Proceedings of the Third ACM Symposium on Cloud Computing. New York, USA: ACM, 2012: 7.

DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters [J]. Communications of the ACM, 2008, 51(1): 107-113.

李建江, 崔健, 王聃, 等. MapReduce 并行编程模型研究综述 [J]. 电子学报, 2011, 39(11): 2635-2642.

LI Jianjiang, CUI Jian, WANG Dan, et al. Survey of MapReduce parallel programming model [J]. Acta Electronica Sinica, 2011, 39(11): 2635-2642.

ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi-user MapReduce clusters, UCB/EECS-2009-55 [R]. Berkeley, CA, USA: University of California. EECS Department, 2009.

YANG Hailong, LUAN Zhongzhi, LI Wenjun, et al. MapReduce workload modeling with statistical approach [J]. Journal of Grid Computing, 2012, 10(2): 279-310.

HUANG Shengsheng, HUANG Jie, DAI Jinquan, et al. The HiBench benchmark suite: characterization of the MapReduce-based data analysis [C]∥Proceedings of the 2010 IEEE 26th International Conference on Data Engineering Workshops. Berlin, Germany: Springer, 2010: 41-51.

ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environments [C]∥Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX, 2008: 29-42.

KWON Y C, BALAZINSKA M, HOWE B, et al. Skewtune: mitigating skew in MapReduce applications [C]∥Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. New York, USA: ACM, 2012: 25-36.

RASOOLI A, DOWN D G. A hybrid scheduling approach for scalable heterogeneous Hadoop systems [C]∥Proceedings of the 2012 SC Companion on High Performance Computing, Networking, Storage and Analysis. Piscataway, NJ, USA: IEEE, 2012: 1284-1291.

TIAN Chao, ZHOU Haojie, HE Yongqiang, et al. A dynamic MapReduce scheduler for heterogeneous workloads [C]∥Proceedings of the 2009 8th International Conference on Grid and Cloud Computing. Piscataway, NJ, USA: IEEE, 2009: 218-224.

LU Peng, LEE Y C, WANG Chen, et al. Workload characteristic oriented scheduler for MapReduce [C]∥Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems. Piscataway, NJ, USA: IEEE, 2012: 156-163.

TANG Shanjiang, LEE Bu-Sung, HE Bingsheng. Dynamic slot allocation technique for MapReduce clusters [C]∥Proceedings of the 2013 IEEE International Conference on Cluster Computing. Piscataway, NJ, USA: IEEE, 2013: 1-8.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于思维进化的集群作业调度方法研究

云计算环境下的大规模图状数据处理任务调度算法

云环境下基于多属性层次分析的虚拟机部署与调度策略

云计算环境中的组合文档模型及其访问控制方案

一种多租户云的内部网络共享策略