VPN Connection

Fig. 10.8  Example of a simple Cloud seeding scenario

infrastructure in order to have access to computing resources with a specific configuration such as GPU equipment. This type of requirement can also happen in cases where some tasks use software protected by a license, and use some private data or algorithms which should never get out to an untrusted environment such as a Cloud.

Cloud Seeding aims at providing a solution to such problems by extending the capabilities of a Cloud with specific external resources.

Figure 10.8 shows a simple Cloud seeding example. In this scenario, most parts of the application are hosted in the Cloud. The ProActive Scheduler and the Resource Manager are hosted on an Amazon EC2 instance, as well as the computing nodes that are used by the scheduler. However, some tasks need a particular resource configuration in order to be executed, such as a GPU processor. The resource manager can handle, in addition to the Amazon EC2 nodes, a set of special nodes from the customer network gathered in a seed subnet.

As seen in Section 3.3, multiple technical solutions can be used to build such a configuration. In our example, we used a VPN-based solution. To enable the scheduler and the resource manager to communicate with these special nodes, we gather them in a seed subnet that hosts a VPN gateway and connects the scheduler and the resource manager to this VPN gateway. However, this type of configuration does not allow Amazon EC2 instances to communicate directly with these special nodes. If we want to permit such communication, one solution is for each Amazon EC2 instance to create a VPN connection with the VPN gateway. Another solution is to build an Amazon VPC, as described in Section 3.3.2, to connect the seed and the VPC subnets together, thus creating a virtual network authorizing communication between any nodes in this network.

10.6  Conclusion

In this paper, we have evaluated the benefits of Cloud computing for scientific applications. Although the performance can be similar to a dedicated cluster for computationally-intensive code, it drops when running communication-intensive code. This observation motivates the need for mixing Cloud and traditional computing platforms. Hybrid platforms require mechanisms adapted to gather resources, deploy applications, and ensure efficient communications.

Although a low-level solution such as VPN may be useful, these solutions are limited because they hide the heterogeneity and prevent the adaptation of the deployment and applications to the environment. Using ProActive and the ProActive/GCM deployment allows specifying how resources are acquired and how the communication should be performed (through simple protocols or message tunneling and forwarding).

We have shown that these mechanisms are powerful enough to build a modular grid/Cloud middleware to support scientific domain-decomposition applications. We illustrated this through adaptating a complex communicationand networkintensive HPC application to run efficiently over a mix of Cloud resources and dedicated ones, without much overhead. Finally, we have also shown how ProActive Resource Manager enables dynamic mixing of Cloud and grid platforms, allowing both Cloud bursting and Cloud seeding within a single framework. These mechanisms also offer a solution to smoothly migrating applications from clusters and grids to Clouds.

Experiments presented in this paper were carried out using the Grid’5000 experimental testbed, developed under the INRIA ALADDIN development action with support from CNRS, RENATER, and several French Universities, as well as other funding bodies (see https://www.grid5000.fr). The authors would also like to thank Amazon Web Services and the PacaGrid CPER for providing computing resources.

References

1.Nimbus toolkit. http://workspace.globus.org/

2.Opennebula project. http://www.opennebula.org/

3.Proactive parallel suite. http://proactive.inria.fr.

4.Baude F, Caromel D, Mestre L, Huet F, Vayssière J (2002) Interactive and descriptor-based deployment of object-oriented grid applications. Proceedings of the 11th IEEE international symposium on high performance distributed computing

5.Bernacki M, Lanteri S, Piperno S (2006) Time-domain parallel simulation of heterogeneous wave propagation on unstructured grids using explicit non-diffusive, discontinuous Galerkin methods. J Comp Acoustics 14(1):57–81

6.enStratus. The enstratus framework for amazon ec2, http://www.enstratus.com/

7.E. Grid. Elastic grid. http://www.elastic-grid.com/

8.GridGain. Grid grain: The open cloud platform. http://www.gridgain.com/

9.Mathias E, Cavé V, Lanteri S, Baude F (2009) Grid-enabling spmd applications through hierarchical partitioning and a component-based runtime. In: Proceedings of the 15th international euro-par conference on parallel processing (Euro-Par ’09), Springer-Verlag, Berlin, Heidelberg, pp 691–703

10. Napper J, Bientinesi P (2009) Can cloud computing reach the top500? In: Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshop (UCHPC-MAW ’09), ACM, New York, USA, pp 17–20

11. Scalr. The scalr framework for cloud computing, http://www.scalr.net/

12. Vertebra. Easy to manage framework for orchestrating complex processes in the cloud. http://www.engineyard.com/vertebra.

Chapter 11

Resource Management for Hybrid Grid and Cloud Computing

Simon Ostermann, Radu Prodan, and Thomas Fahringer

Abstract  From its start of using supercomputers, scientific computing constantly evolved to the next levels such as cluster computing, meta-computing, or computational Grids. Today, Cloud Computing is emerging as the paradigm for the next generation of large-scale scientific computing, eliminating the need for hosting expensive computing hardware. Scientists still have their Grid environments in place and can benefit from extending them using leased Cloud resources whenever needed. This paradigm shift opens new problems that need to be analyzed, such as integration of this new resource class into existing environments, applications on the resources, and security. The virtualization overheads for deployment and starting of a virtual machine image are new factors, which will need to be considered when choosing scheduling mechanisms. In this chapter, we investigate the usability of compute Clouds to extend a Grid workflow middleware and show on a real implementation that this can speed up executions of scientific workflows.

11.1  Introduction

In the last decade, Grid computing gained became popular in the field of scientific computing through the idea of distributed resource sharing among institutions and scientists. Scientific computing is traditionally a high-utilization workload, with production Grids often running at over 80% utilization [1] (generating high and often unpredictable latencies), and with smaller national Grids offering a rather limited amount of high-performance resources. Running large-scale simulations in such overloaded Grid environments often becomes latency-bound or suffers from well-known Grid reliability problems [2].

S. Ostermann (*)

Institute of Computer Science, University of Innsbruck, Technikerstra§e 21a, 6020, Innsbruck, Austria

e-mail: simon@dps.uibk.ac

Systems and Applications, Computer Communications and Networks,

DOI 10.1007/978-1-84996-241-4_11, © Springer-Verlag London Limited 2010