Skip to main content
Journal of Internet Services and Applications
Download PDF
  • SI: Cloud Computing
  • Open access
  • Published:

HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds

Journal of Internet Services and Applications volume 2, pages 207–227 (2011)Cite this article

Abstract

Workflows have been used to represent a variety of applications involving high processing and storage demands. As a solution to supply this necessity, the cloud computing paradigm has emerged as an on-demand resources provider. While public clouds charge users in a per-use basis, private clouds are owned by users and can be utilized with no charge. When a public cloud and a private cloud are merged, we have what we call a hybrid cloud. In a hybrid cloud, the user has elasticity provided by public cloud resources that can be aggregated to the private resources pool as necessary. One question faced by the users in such systems is: Which are the best resources to request from a public cloud based on the current demand and on resources costs? In this paper we deal with this problem, presenting HCOC: The Hybrid Cloud Optimized Cost scheduling algorithm. HCOC decides which resources should be leased from the public cloud and aggregated to the private cloud to provide sufficient processing power to execute a workflow within a given execution time. We present extensive experimental and simulation results which show that HCOC can reduce costs while achieving the established desired execution time.

References

  1. The NIST definition of cloud computing 15. Tech rep, National Institute of Standards and Technology (NIST) (2009). http://csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc

  2. Abels T, Dhawan P, Chandrasekaran B An overview of xen virtualization (2009). http://www.dell.com/downloads/global/power/ps3q05-20050191-Abels.pdf

  3. Amir Y, Awerbuch B, Barak A, Borgstrom RS, Keren A (2000) An opportunity cost approach for job assignment in a scalable computing cluster. IEEE Trans Parallel Distrib Syst 11(7):760–768

    Article  Google Scholar 

  4. Annis J, Zhao Y, Voeckler J, Wilde M, Kent S, Foster I (2002) Applying Chimera virtual data concepts to cluster finding in the Sloan Sky Survey. In: Supercomputing ’02: Proceedings of the 2002 ACM/IEEE conference on supercomputing. IEEE Computer Society Press, Los Alamitos, pp 1–14

    Google Scholar 

  5. Batista DM, da Fonseca NLS, Miyazawa FK (2007) A set of schedulers for grid networks. In: SAC ’07: Proceedings of the ACM symposium on applied computing. ACM, Seoul pp 209–213

    Chapter  Google Scholar 

  6. Bittencourt LF, Madeira ERM (2008) A performance-oriented adaptive scheduler for dependent tasks on grids. Concurr Comput 20(9):1029–1049

    Article  Google Scholar 

  7. Bittencourt LF, Madeira ERM (2010) Towards the scheduling of multiple workflows on computational grids. J Grid Comput 8:419–441. doi:10.1007/s10723-009-9144-1

    Article  Google Scholar 

  8. Bittencourt LF, Senna CR, Madeira ERM (2010) Enabling execution of service workflows in grid/cloud hybrid systems. In: International workshop on cloud management (Cloudman). IEEE Computer Society Press, Osaka

    Google Scholar 

  9. Bittencourt LF, Senna CR, Madeira ERM (2010) Scheduling service workflows for cost optimization in hybrid clouds. In: 6th international conference on network and service management (CNSM). Toronto, Canada, 2010

    Google Scholar 

  10. Van den Bossche R, Vanmechelen K, Broeckhove J (2010) Cost-optimal scheduling in hybrid IaaS clouds for deadline constrained workloads. In: 3rd international conference on cloud computing, pp 228–235

    Google Scholar 

  11. Corrêa RC, Ferreira A, Rebreyend P (1996) Integrating list heuristics into genetic algorithms for multiprocessor scheduling. In: SPDP ’96: Proceedings of the 8th IEEE symposium on parallel and distributed processing (SPDP ’96). IEEE Computer Society, Washington, p 462

    Chapter  Google Scholar 

  12. Deelman E (2008) Clouds: an opportunity for scientific applications? (keynote in the 2008 Cracow Grid Workshops)

  13. Deelman E, Singh G, Su MH, Blythe J, Gil Y, Kesselman C, Mehta G, Vahi K, Berriman GB, Good J, Laity A, Jacob JC, Katz DS (2005) Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci Program 13(3):219–237

    Google Scholar 

  14. Dogan A, Özgüner F (2005) Biobjective scheduling algorithms for execution time-reliability trade-off in heterogeneous computing systems. Comput J 48(3):300–314

    Article  Google Scholar 

  15. Dong F, Akl SG (2006) Scheduling algorithms for grid computing: state of the art and open problems. Tech rep, Queen’s University School of Computing, Kingston, Canada

  16. Fujimoto N, Hagihara K (2003) Near-optimal dynamic task scheduling of precedence constrained coarse-grained tasks onto a computational grid. In: 2nd intl symp on parallel and distributed computing, Slovenia. IEEE Press, New York, pp 80–87

    Google Scholar 

  17. Grajcar M (1999) Genetic list scheduling algorithm for scheduling and allocation on a loosely coupled heterogeneous multiprocessor system. In: DAC ’99: Proceedings of the 36th ACM/IEEE conference on design automation. ACM Press, New York, pp 280–285

    Chapter  Google Scholar 

  18. Hou ESH, Ansari N, Ren H (1994) A genetic algorithm for multiprocessor scheduling. IEEE Trans Parallel Distrib Syst 5(2):113–120

    Article  Google Scholar 

  19. Yu J, Buyya R, Ramamohanarao K (2008) Workflow scheduling algorithms for grid computing. Stud Comput Intell 146:173–214

    Article  Google Scholar 

  20. Kwok YK, Ahmad I (1996) Dynamic critical-path scheduling: an effective technique for allocating task graphs to multiprocessors. IEEE Trans Parallel Distrib Syst 7(5):506–521

    Article  Google Scholar 

  21. Kwok YK, Ahmad I (1999) Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Comput Surv 31(4):406–471

    Article  Google Scholar 

  22. Lindley CA (1991) Practical image processing in: C: acquisition, manipulation and storage: hardware, software, images and text. Wiley, New York

    Google Scholar 

  23. Mendell G (2008) Use of Condor by the LIGO scientific collaboration. In: Padaryn/Condor Week. Wisconsin, USA

    Google Scholar 

  24. Nurmi D, Wolski R, Grzegorczyk C, Obertelli G, Soman S, Youseff L, Zagorodnov D (2009) The Eucalyptus open-source cloud-computing system. In: 2009 9th IEEE/ACM international symposium on cluster computing and the grid (CCGRID). IEEE Press, Washington, pp 124–131

    Chapter  Google Scholar 

  25. Phillips C, Stein C, Wein J (1997) Task scheduling in networks. SIAM J Discrete Math 10(4):573–598

    Article  MATH  MathSciNet  Google Scholar 

  26. Pinedo ML (2008) Scheduling: theory, algorithms, and systems. Springer, Berlin

    Google Scholar 

  27. Rahman M, Venugopal S, Buyya R (2007) A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: Third IEEE international conference on e-Science and grid computing. IEEE Computer Society, Washington, pp 35–42

    Google Scholar 

  28. Ramakrishnan A, Singh G, Zhao H, Deelman E, Sakellariou R, Vahi K, Blackburn K, Meyers D, Samidi M (2007) Scheduling data-intensive workflows onto storage-constrained distributed resources. In: CCGRID ’07: Proceedings of the seventh IEEE international symposium on cluster computing and the grid. IEEE Computer Society, Washington, pp 401–409

    Chapter  Google Scholar 

  29. Sakellariou R, Zhao H (2004) A hybrid heuristic for DAG scheduling on heterogeneous systems. In: 13th heterogeneous computing workshop. IEEE Computer Society, Washington, pp 111–123

    Google Scholar 

  30. Senna CR, Bittencourt LF, Madeira ERM (2009) Execution of service workflows in grid environments. In: International conference on testbeds and research infrastructures for the development of networks & communities (Tridentcom). IEEE Computer Society, Washington

    Google Scholar 

  31. Smith JE, Nair R (2005) The architecture of virtual machines. Computer 38(5):32–38

    Article  Google Scholar 

  32. Topcuoglu H, Hariri S, Wu MY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  33. Yang T, Gerasoulis A (1994) DSC: scheduling parallel tasks on an unbounded number of processors. IEEE Trans Parallel Distrib Syst 5(9):951–967

    Article  Google Scholar 

  34. Yu J, Buyya R (2006) Scheduling scientific workflow applications with deadline and budget constraints using genetic algorithms. Sci Program 14(3–4), 217–230

    Google Scholar 

  35. Yu J, Buyya R, Tham CK (2005) Cost-based scheduling of scientific workflow application on utility grids. In: Proceedings of the first international conference on e-science and grid computing. IEEE Computer Society, Washington, pp 140–147

    Google Scholar 

  36. Zhang Q, Cheng L, Boutaba R (2010) Cloud computing: state-of-the-art and research challenges. J Internet Serv Appl 1(1):7–18

    Article  Google Scholar 

  37. Zhang S, Zong Y, Ding Z, Liu J (2005) Workflow-oriented grid service composition and scheduling. In: International symposium on information technology: coding and computing, v 2, April 2005, Las Vegas, Nevada, USA. IEEE Computer Society, Washington, pp 214–219

    Google Scholar 

  38. Zhao Y, Dobson J, Foster I, Moreau L, Wilde M (2005) A notation and system for expressing and executing cleanly typed workflows on messy scientific data. SIGMOD Rec 34(3):37–43

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computing, University of Campinas, Av. Albert Einstein, 1251 Cidade Universitária, 13083-852, Campinas, São Paulo, Brazil

    Luiz Fernando Bittencourt & Edmundo Roberto Mauro Madeira

Authors
  1. Luiz Fernando Bittencourt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edmundo Roberto Mauro Madeira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luiz Fernando Bittencourt.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Bittencourt, L.F., Madeira, E.R.M. HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J Internet Serv Appl 2, 207–227 (2011). https://doi.org/10.1007/s13174-011-0032-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13174-011-0032-0

Keywords