A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation

doi:10.7652/xjtuxb201508012

您当前的位置：

首页 >

文章列表页 >

A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation

更新时间：2025-07-09

- A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation
- Vol. 49, Issue 8, Pages: 69-74+108(2015)
- 作者机构：
  
  西安交通大学电子与信息工程学院,西安,710049
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb201508012
  CLC： TP391
- Online First：10 August 2015，
  
  Published：2015
- 稿件说明：
移动端阅览
A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation[J]. 2015, 49(8): 69-74+108.
DOI：

A Resource License Scheduling Method for Hadoop in Cloud Computing Using Resource Allocation[J]. 2015, 49(8): 69-74+108. DOI： 10.7652/xjtuxb201508012.

摘要

针对云计算环境中Hadoop平台由于节点计算能力差异、多样混合负载共存等原因而出现的性能不佳的问题

提出一种采用资源划分的资源许可方法。该方法在云计算环境下通过减少资源浪费或负载过重等情况的出现来提高系统性能。该方法采集资源信息并推测任务资源需求

根据可用计算资源和任务需求动态划分、调度资源; 使用与资源无耦合的资源许可启动任务并控制任务数量调节资源利用率以适应云环境。使用该方法对比公平调度器在国家高性能计算中心(西安)进行测试发现:单作业在资源竞争环境中优于公平调度器的静态结果; 混合负载在3种测试环境中完成时间分别平均减少了27.5%、37.1%和50.98%

性能显著提升。实验结果表明

该方法可以适应负载资源需求和可用计算资源的变化

灵活划分计算资源

解决Hadoop在云环境中的性能不佳问题。

Abstract

A resource license scheduling method using resource allocation is proposed to improve the poor performance caused by different computing capacities of nodes and mixed workload in cloud computing. The method improves the performance through reducing the resources wasting or overload. It collects resource information and estimates resource requirements of workloads and allocates computing resources dynamically according to available resources and resource requirements of workloads. Licenses uncoupled with resources are used to launch tasks and to adjust the number of parallel tasks to adapt the cloud environment by controlling the number of licenses. The method is evaluated in the national high performance computing center(Xi'an). Results show that the completion times of single job workloads of the proposed method are better than those of the FAIR scheduler in competitive environments. Moreover

the completion times of mixed workloads of the method in three environments reduce 27.5%

37.1% and 50.98% respectively on average

that is

the performance of the method has a significant improvement. It can be concluded from the results that the proposed method adapts the complex environment and solve the performance problem in cloud computing.

关键词

Keywords

references

REISS C, TUMANOV A, GANGER G R, et al. Heterogeneity and dynamicity of clouds at scale: google trace analysis [C]∥Proceedings of the Third ACM Symposium on Cloud Computing. New York, USA: ACM, 2012: 7.

DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters [J]. Communications of the ACM, 2008, 51(1): 107-113.

李建江, 崔健, 王聃, 等. MapReduce 并行编程模型研究综述 [J]. 电子学报, 2011, 39(11): 2635-2642.

LI Jianjiang, CUI Jian, WANG Dan, et al. Survey of MapReduce parallel programming model [J]. Acta Electronica Sinica, 2011, 39(11): 2635-2642.

ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi-user MapReduce clusters, UCB/EECS-2009-55 [R]. Berkeley, CA, USA: University of California. EECS Department, 2009.

YANG Hailong, LUAN Zhongzhi, LI Wenjun, et al. MapReduce workload modeling with statistical approach [J]. Journal of Grid Computing, 2012, 10(2): 279-310.

HUANG Shengsheng, HUANG Jie, DAI Jinquan, et al. The HiBench benchmark suite: characterization of the MapReduce-based data analysis [C]∥Proceedings of the 2010 IEEE 26th International Conference on Data Engineering Workshops. Berlin, Germany: Springer, 2010: 41-51.

ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environments [C]∥Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX, 2008: 29-42.

KWON Y C, BALAZINSKA M, HOWE B, et al. Skewtune: mitigating skew in MapReduce applications [C]∥Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. New York, USA: ACM, 2012: 25-36.

RASOOLI A, DOWN D G. A hybrid scheduling approach for scalable heterogeneous Hadoop systems [C]∥Proceedings of the 2012 SC Companion on High Performance Computing, Networking, Storage and Analysis. Piscataway, NJ, USA: IEEE, 2012: 1284-1291.

TIAN Chao, ZHOU Haojie, HE Yongqiang, et al. A dynamic MapReduce scheduler for heterogeneous workloads [C]∥Proceedings of the 2009 8th International Conference on Grid and Cloud Computing. Piscataway, NJ, USA: IEEE, 2009: 218-224.

LU Peng, LEE Y C, WANG Chen, et al. Workload characteristic oriented scheduler for MapReduce [C]∥Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems. Piscataway, NJ, USA: IEEE, 2012: 156-163.

TANG Shanjiang, LEE Bu-Sung, HE Bingsheng. Dynamic slot allocation technique for MapReduce clusters [C]∥Proceedings of the 2013 IEEE International Conference on Cluster Computing. Piscataway, NJ, USA: IEEE, 2013: 1-8.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on Mind Evolutionary Computation Based Job Scheduling for Server Clusters

A Task Scheduling Algorithm for Large Graph Processing Cloud in Computing

Deployment and Scheduling of Virtual Machines in Cloud Computing: An “AHP” Approach

A Composite Document Model and Its Access Control Scheme in Cloud Computing

An Intra-Network Sharing Policy for Multi-Tenant Clouds

Related Author

伍卫国

董小社

刘伟哲

薛正华

苏森

刘一阳

黄庆佳

李健

Related Institution

Department of Computer Science and Technology, Xi'an Jiaotong University

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications

Shaanxi Province Key Laboratory of Computer Network, Xi'an Jiaotong University

School of Computer Science and Technology, Xidian University

Faculty of Software, Fujian Normal University

⁰