11.4  Evaluation

Table 11.3  Overview of resources used from the grid and the private cloud for workflow execution

Table 11.4  Wien2K execution time and cost analysis on the Austrian grid with and without cloud resources for different number of parallel activities and problem sizes

machine size of each Grid site to 12 cores to eliminate the variability in the resource availability and make the results across different experiments comparable.

We used a just-in-time scheduling mechanism that tries to map each activity onto the fastest available Grid resource. Once the Grid becomes full (because the size of the workflow parallel loops is larger than the total number of cores in the testbed), the scheduler starts requesting additional Cloud resources for executing, in parallel, the remaining workflow activities. Once these additional resources are available, they will be used to link Grid resources with different job submission methods.

Our goal was to compare the workflow execution for different problem sizes on the four Grid sites, with the execution using the same Grid environment supplemented by additional Cloud resources from dps.cloud. We executed each workflow instance five times and reported the average values obtained. The runtime variability in the Austrian Grid was less than 5%, because the testbed was idle during our experiments and each CPU was dedicated to running its activity with no external load or other queuing overheads.

Table11.4 shows the workflow execution times for 376 and 193 parallel activities in six different configurations. The small, medium, and big configuration values represent a problem size parameter that influences the execution time of the parallel activities. The improvement in using Cloud resources when compared with using only the four Grid sites increases from a small 1.08 speedup for short workflows with 14-min execution time, to a good 1.67 speedup for large workflows with 93-min execution time. The results show that a small and rather short workflow does not benefit much from the Cloud resources due to the high ratio between the smaller

computation and the high provisioning and data transfer overheads. The main bottleneck­ when using Cloud resources is that the provisioned single core instances use separate file systems that require separate file transfers to start the computation. In contrast, Grid sites are usually parallel machines that share one file system across a larger number of cores, which significantly decreases the data transfer overheads. Nevertheless, for large problem sizes, the Cloud resources can help to significantly shorten the workflow completion time in case Grids become overloaded.

Table 11.5 gives further details on the file transfer overheads and the distribution of activity instances between the pure Grid and the combined Grid-Cloud execution. The file transfer overhead can be reduced by increasing the size of a resource class (i.e. number of cores underneath one instance, which share a file system and the input files for execution), which may result in a lower resource allocation efficiency as the resource allocation granularity increases. We plan to investigate this tradeoff in future work.

To understand and quantify the benefit and the potential costs of using commercial Clouds for similar experiments (without running the Wien2k workflows once again because of cost reasons), we executed the LINPACK benchmark [26] that measures the GFlop sustained performance of the resource classes offered by three Cloud providers: Amazon EC2, GoGrid (GG), and our academic dps.cloud (see Table 11.1). We configured LINPACK to use the GotoBLAS linear algebra library (one of the fastest implementations on Opteron processors in our experience) and MPI Chameleon [27] for instances with multiple cores. Table 11.6 summarizes the results that show the m1.large EC2 instance as being the closest to the dps.cloud, assuming that the two cores are used separately, which indicates an approximate realistic cost of $0.20 per core hour. The best sustained performance is offered by GG; however, it has extremely large resource provisioning latencies

Table 11.5  Grid versus cloud file transfer and activity instance distribution­ to grid and cloud resources [t]

Table 11.6  Average LINPACK sustained performance and resource provisioning latency results of various resource classes (see Table 11.1)