148 |
L. Shang et al. |
9.2 Overview of YML
YML is a workflow-based framework dedicated to execution of parallel applications on various middleware. Now it can support two middlewares: OmniRPC [19] and XtremWeb[20]. OmniRPC can harness dedicated computing resources in cluster and grid environments, while XtremWeb can collect volunteer computing resources in Desktop Grid. Condor is on the way to being integrated into YML. Figure 9.1 shows us the overview of YML.
There are four parts in the Fig. 9.1, which are CLIENT, Data Repository, YML, and YML workers.
CLIENT provides the end-users an intuitive way to express parallel applications by means of a workflow. The description language of YML is called YvetteML and can express several execution structures of parallel programs, such as sequential execution, parallel execution, and conditional branch and event notification/reception (event signals are used to control when an operation can be executed). A simple example can be presented through Fig. 9.2. What we want to emphasize here is that end-users just use YvetteML to describe the workflow and he/she must not know how to program using special programming languages (e.g. Java, C, C++, Fortune, MPI). The parallel program described using YvetteML can be run on different platforms without any change. We will explain this point in the part of YML workers in detail.
|
|
CLIENT |
|
|
|
|
application of the client |
||
|
|
www |
|
|
DATA Repository |
|
YML |
|
|
development catalog |
|
FRONT-END Layer |
||
component |
YML |
YML |
||
|
||||
resources informations |
generator |
compiler |
scheduler |
|
|
|
|
||
execution catalog |
|
BACKENT-END Layer |
||
|
|
|
||
|
binary |
backend |
middleware |
|
|
generator |
|||
data repository server |
scheduler |
specific backend |
||
|
||||
|
|
|
||
www
YML WORKER middlewares (yml workers)
Fig. 9.1 Overview of YML framework [17]
9 A Reference Architecture Based on Workflow for Building Scientific Private Clouds |
149 |
|||||
|
|
A |
|
|
Sequential |
|
|
|
|
|
|
A |
|
|
|
|
|
|
// |
|
|
|
|
|
|
Parallel |
|
B |
B |
B |
|
B |
B |
|
|
|
|
|
|
|
|
|
|
|
|
|
End Parallel |
|
|
|
|
|
|
// |
|
|
|
C |
|
|
C |
|
|
|
|
|
|
// |
|
|
|
|
|
|
Parallel |
|
|
D |
D |
D |
|
D |
|
|
|
|
|
|||
End Parallel
|
// |
|
E |
E |
End Sequential |
|
|
Fig. 9.2 Sketch of high-level program interface |
|
Data Repository contains four parts, which are Development catalog, Resources information, Execution catalog, and Data server. Development catalog has functions (e.g. function A, B in Fig. 9.2.) developed using C or C++. These functions can be reused in other applications. For example, application 1 and application 2 both need “function A” and if you develop “function A” for application 1, you can reuse it in application 2 without any change. After “function A” in Development catalog is registered in YML, “function A” will become the executable function in the Execution catalog. Resources information is information about which computing nodes can be utilized. The information is dynamically changed according to the availability of computing resources. As well known to us all, the key characteristic of volunteer computing resources is volatility. So, accurate prediction of volunteer computing resources is very important and it is also the key research point of this paper. Data server is used to store data, and its transport model is based on server/ worker. So, adopting server/worker-based data transport model is another key research point of this paper.
YML is the core of the “sandwich” architecture of YML framework. It can hide the complexity and heterogeneity of the underlying computing platforms and provide a unique programming interface to end-users. YML has two layers, which are Front-end and Back-end. Front-end can parse the pseudocode-based program in CLIENT and invoke executable functions accordingly to form an executable graph. According to this graph, YML scheduler will allocate tasks to Back-end scheduler. Then, Back-end scheduler allocates those tasks to different computing resources according to Resources information. Back-end provides different interfaces to different middleware.