A Delay Scheduling Algorithm for Short Jobs

doi:10.7652/xjtuxb201502001

您当前的位置：

首页 >

文章列表页 >

A Delay Scheduling Algorithm for Short Jobs

更新时间：2025-07-09

- A Delay Scheduling Algorithm for Short Jobs
- Vol. 49, Issue 2, Pages: 1-5+54(2015)
- 作者机构：
  
  西安交通大学电子与信息工程学院,西安,710049
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb201502001
  CLC： TP391
- Online First：10 February 2015，
  
  Published：2015
- 稿件说明：
移动端阅览
A Delay Scheduling Algorithm for Short Jobs[J]. 2015, 49(2): 1-5+54.
DOI：

A Delay Scheduling Algorithm for Short Jobs[J]. 2015, 49(2): 1-5+54. DOI： 10.7652/xjtuxb201502001.

摘要

针对短作业场景下YARN平台中延迟调度算法基于静态时间等待阈值

不能进行合理等待的问题

提出了一种云计算环境中基于本地性资源预测的延迟调度算法(locality resource forecast delay scheduling

LRFD)。该算法综合考虑短作业和资源可用性动态变化的特点进行任务调度

根据节点上任务的完成进度和作业未处理数据在集群中的分布状况预估作业的本地性资源信息

从而判断是否需要进行等待以提高系统性能

实现了对本地性资源的合理等待。实验结果表明:在短作业场景下

LRFD算法的性能和稳定性均优于已有的延迟算法

作业性能平均提升约10%

最大加速比可达3倍以上。

Abstract

A delay scheduling algorithm based on locality resource forecasting(LRFD)is proposed to address the unreasonable waiting problem generalized by the static time-wait threshold in delay scheduling algorithm of YARN platform for short jobs. The algorithm takes both the characteristics of short jobs and dynamic resource availability into consideration to assign tasks. It estimates local resources to make reasonable waiting according to both the task progress on the nodes and the unhandled splits distribution in the cluster of the job. Experimental results and comparison with the traditional delay scheduling algorithm show that LRFD gets a better stability and improves the performance about 10% for short jobs on average and achieves a maximum speedup up to three times.

关键词

Keywords

references

DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters [J]. Communications of the ACM, 2008, 51(1): 107-113.

CHEN Y, ALSPAUGH S, KATZ R. Interactive analytical processing in big data systems: a cross-industry study of MapReduce workloads [J]. Proceedings of the Very Large Data Base Endowmat, 2012, 5(12): 1802-1813.

ISARD M, PRABHAKARAN V, CURREY J, et al. Quincy: fair scheduling for distributed computing clusters [C] ∥Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. New York, NY, USA: ACM, 2009: 261-276.

REN Z, XU X, WAN J, et al. Workload analysis, implications, and optimization on a production hadoop cluster: a case study on Taobao [J]. IEEE Transactions on Services Computing, 2014, 7(2): 307-321.

ZAHARIA M, CHOWDHURY M, FRANKLIN M J, et al. Spark: cluster computing with working sets [C]∥Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. Berkeley, CA, USA: USENIX Association, 2010: 10.

HENDERSON R L. Job scheduling under the portable batch system [C]∥Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing. Berlin, Germany,: Springer-Verlag, 1995: 279-294.

FREY J, TANNENBAUM T, LIVNY M, et al. Condor-G: a computation management agent for multi-institutional grids [J]. Cluster Computing, 2002, 5(3): 237-246.

HINDMAN B, KONWINSKI A, ZAHARIA M, et al. Mesos: a platform for fine-grained resource sharing in the data center [C]∥Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2011: 295-308.

VAVILAPALLI V K, MURTHY A C, DOUGLAS C, et al. Apache hadoop yarn: yet another resource negotiator [C]∥Proceedings of the 4th Annual Symposium on Cloud Computing. New York, NY, USA: ACM, 2013: 1-16.

ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi-user Mapreduce clusters[R]. Berkeley, California, USA: EECS Department, University of California, Berkeley, 2009: 1-16.

ZAHARIA M, BORTHAKUR D, SEN SARMA J, et al. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling [C]∥Proceedings of the 5th European Conference on Computer Systems, New York, NY, USA: ACM, 2010: 265-278.

ELMELEEGY K. Piranha: optimizing short jobs in hadoop [J]. Proceedings of the Very Large Data Base Endowment, 2013, 6(11): 985-996.

ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environments [C]∥Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2008: 29-42.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A Task Scheduling Algorithm for Large Graph Processing Cloud in Computing

Deployment and Scheduling of Virtual Machines in Cloud Computing: An “AHP” Approach

A Composite Document Model and Its Access Control Scheme in Cloud Computing

An Intra-Network Sharing Policy for Multi-Tenant Clouds

Multiplexing of Backup Virtual Machine Based on Single-Step Heuristic Policy

Related Author

苏森

刘一阳

黄庆佳

李健

庄威 1

桂小林 1

林建材 1

Related Institution

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications

Shaanxi Province Key Laboratory of Computer Network, Xi'an Jiaotong University

School of Computer Science and Technology, Xidian University

Faculty of Software, Fujian Normal University

National Network New Media Engineering Research Center, Institute of Acoustics, Chinese Academy of Sciences

⁰