IT-SC book: Advanced CORBA® Programming with C++

In Section 11.11 we explain the operations that the ORB provides to allow you to control how it handles events such as client connection initiation and the arrival of requests on its network connections. Some applications use a blocking event handling model in which the application main hands control of the main thread over to ORB::run. Others use a non-blocking event handling model in which they temporarily give the main thread to the ORB to perform a unit of work.

Some applications contain event handling loops for software other than the ORB, such as for a windowing system. For such applications that are multithreaded, there are three approaches for integrating these disparate event loops.

Use the non-blocking ORB::work_pending and ORB:: perform_work operations to control the ORB's events. Mix them together in your own event handling loop that also uses non-blocking event handling for the other software. Section 11.11.2 shows an example that integrates non-blocking ORB event handling together with the event loop of a hypothetical GUI library.

Create a separate thread for each event loop. Because the CORBA specification clearly states that portable applications must yield the main thread to the ORB to allow it to handle its events, you must run the event loops of the other software in threads other than the main thread. If this is not possible because of either the nature of the other software or the multithreading support provided by the underlying platform, you should fall back to using the approach just described.

Collect all the file descriptors used by all the software packages that you are integrating, including your ORB. Then either write your own select-based listening code to handle them or nominate one of the software packages to monitor all of them. When an event occurs on one of the file descriptors, you invoke its associated software package and tell it to handle its file descriptor event.

Although this approach works well for many applications, it requires that you invoke proprietary functions on your ORB to get its file descriptors. It also assumes that all transports used by the ORB are based on file descriptors, and that is not always the case (such as in an embedded system in which the "transport" is actually a hardware backplane). Furthermore, it makes it more difficult for the ORB to manage its own connections, and that could hurt your application's scalability. We therefore recommend that you avoid this approach unless it is absolutely necessary.

Whether you use the first or the second approach depends on the nature of your application. The second approach allows for more parallelism than the first, but it is also harder to get right because of the need to ensure that all code throughout the application is thread-safe. Note that the first approach works for both single-threaded and multithreaded applications.

21.5 Multithreading Strategies

837

IT-SC book: Advanced CORBA® Programming with C++

No matter how the ORB you use implements multithreaded request dispatch, your servants must be able to cope with concurrent invocations, possibly even for the same method. This means that you must develop a locking strategy to ensure that access to object state and shared data structures is interlocked correctly.

There are basically two locking strategies that you can use: coarse-grained and finegrained. With the coarse-grained approach, servants deal only with a single thread of control at a time. In other words, only one thread is ever running within a single servant at any point. The fine-grained approach allows multiple threads of control to be present in a servant simultaneously. The difference between the two models is one of locking granularity.

You can use the coarse-grained model by simply locking a per-servant mutex at the beginning of every method call and unlocking it at the end. This technique automatically protects any per-servant state against concurrent access. However, you must also make sure that any state that is shared between servants is also protected by a separate lock. This approach is fairly easy to implement and maintain, and it suffices for many applications.

With the fine-grained model, each piece of per-servant and shared state is protected by its own mutex. Because locking is done at a much finer granularity than for the coarsegrained model, more parallelism is possible because of reduced lock contention. On the other hand, if each method tends to access all pieces of per-servant and shared state, this model can be less efficient because of its greater locking overhead. It is also much harder to implement this model correctly because you must make sure that you use the right lock for the right state and that you always acquire locks in the same order to prevent deadlock.

21.6 Implementing a Multithreaded Server

In this section, we explain how to deal with server application concurrency issues by making our climate control system capable of running in a multithreaded ORB environment. Because difficult concurrency problems can crop up when you manage servant life cycles with respect to the objects they incarnate, our example is based on adding thread safety to the CCS object creation and removal operations introduced in Chapter 12. Specifically, we explore multithreading issues for the device creation and removal functions shown in Section 12.6.3 for the servant locator version of the Evictor pattern. We expect that several methods on our servants may be called simultaneously by different threads, so we use the fine-grained multithreading strategy.

Figure 21.1 illustrates the participants in a locator-based evictor implementation. The evictor queue keeps a list of servants in LRU order. Our servant locator adds servants to the queue as it creates them, but it first evicts the LRU servant if the queue has reached its maximum capacity. You may want to revisit Section 12.6 to refresh your understanding of how we implemented the Evictor pattern using a servant locator for the single-threaded case.

838

IT-SC book: Advanced CORBA® Programming with C++

Figure 21.1 Implementing the Evictor pattern using a servant locator.

For synchronization primitives, all examples in this section use the multithreading C++ wrappers that are freely available as part of the ACE toolkit.[1] Even if your ORB implementation supplies its own multithreading wrappers, they are likely to be similar in form and function to the ACE wrappers.

[1] You can obtain directions for downloading the source code for the ACE C++ wrappers from http://www.cs.wustl. edu/~schmidt/ACE-obtain.html.

21.6.1 Review of CCS Life Cycle Operations

In Section 12.3 we show how to add factory operations for creating thermometer and thermostat objects to the CCS::Controller interface.

#pragma prefix "acme.com" module CCS {

// ...

interface Controller {

exception DuplicateAsset {};

Thermometer create_thermometer(

) raises(DuplicateAsset, BadTemp);

// Other operations...

};

};

To create a Thermometer, you invoke create_thermometer on the

Controller object, passing it the asset number and the location of the new

839

IT-SC book: Advanced CORBA® Programming with C++

thermometer. You create Thermostat objects similarly, but their factory operation also requires a value for the initial thermostat temperature setting. After it is created, each Thermometer and Thermostat object is incarnated by a different servant.

The Thermometer interface supplies the remove operation. For the reasons mentioned in Section 12.5.5, we do not inherit remove from the

CosLifeCycle::LifeCy-cleObject interface. When a client invokes remove, the target Thermometer or Thermostat object is destroyed. Any further invocations using the same object reference will cause the OBJECT_NOT_EXIST exception to be raised.

21.6.2 General Application Issues

Our multithreaded evictor implementation example does not show POA creation. You can assume that we have two POAs: the first one, for the singleton Controller object, is a child of the Root POA, and the other one, for the Thermometer and Thermostat objects, is a child of the first POA. We explicitly activate the Controller object, and its POA has the ORB_CTRL_MODEL, PERSISTENT, RETAIN, UNIQUE_ID, and

USE_ACTIVE_OBJECT_MAP_ONLY policy values. We register the servant locator in the POA that supports the Thermometer and Thermostat objects. This second POA has the ORB_CTRL_MODEL, PERSISTENT, NON_RETAIN, UNIQUE_ID, and USE_SERVANT_MANAGER policy values.

Note that putting the Controller object in its own separate POA in this example differs from the single-threaded example in Section 12.6. Because an ORB can concurrently dispatch requests to multiple POAs even if they each have the SINGLE_THREAD_MODEL policy value, the single-threaded example registers all servants under a single POA to prevent concurrent operations on the servants. One result of that design is that the servant locator must recognize and handle the Controller object ID as a special case. In our multithreaded evictor implementation, we handle concurrency explicitly, so we can safely use multiple POAs as described earlier.

Any Thermometer and Thermostat operations that access or modify the state of a device do so by sending messages over the ICP network. We assume that these device operations are atomic, so we do not serialize our access to the ICP network.

Another difference between the single-threaded evictor implementation and the multithreaded version is the data structure that the Controller_impl servant uses to keep track of all the devices. The single-threaded implementation uses an STL map to allow the Controller_impl to associate each servant with the asset number of the device it incarnates. This arrangement allows the Controller_impl to directly delete servants when they are no longer needed. In the multithreaded version, however, servants invoke delete on themselves, as Sections 21.6.7 and 21.6.8 describe. Therefore, the multithreaded Controller_impl keeps only an STL set of device

840

IT-SC book: Advanced CORBA® Programming with C++

asset numbers. If a request arrives for a device that is not in the asset number set, we raise the OBJECT_NOT_EXIST exception.

Just as for the single-threaded evictor implementation, we implement the evictor queue using an STL list:

typedef list<Thermometer_impl *> EvictorQueue;

The evictor queue stores pointers to Thermometer_impl servants. Because

Thermostat_impl derives from Thermometer_impl, the queue can handle servants of both types. The evictor queue instance is a private data member of the servant locator.

The servant locator also keeps track of all the servants that are in use. It uses an STL map for this purpose:

typedef map< CCS::AssetType,

EvictorQueue::iterator > ActiveObjectMap;

The servant locator uses an instance of this type to map between a target object's asset number and the position of its servant, if any, in the evictor queue. Because this map serves a purpose very similar to that of the POA's Active Object Map, we call our map type ActiveObjectMap. However, you should keep in mind that the servant locator's active object map—which, like the evictor queue, is a private data member—is not related to any POA's Active Object Map. (Our POA is a NON_RETAIN POA, so it does not even have an Active Object Map.)

21.6.3 Concurrency Issues

Because the create_thermometer and create_thermostat operations can each be invoked concurrently by different clients, two or more clients may concurrently invoke the same creation operation with the same arguments. This would result in the same object being created multiple times. We therefore must serialize invocations of the create_thermometer and create_thermostat operations.

The factory operations also share data with the Thermometer::remove operation. Specifically, both of the creation operations and the remove operation must access the controller's set of asset numbers to add or erase the asset numbers of their target devices. This means that we must serialize access to the controller's set to prevent it from being updated simultaneously by multiple threads.

The servant locator's preinvoke method does all the work needed to keep the evictor queue in LRU order and to evict servants when necessary. Even though preinvoke is

841

IT-SC book: Advanced CORBA® Programming with C++

the only place in our application where the evictor queue is accessed and modified, we still must address concurrency issues for it. The POA may call preinvoke concurrently from several different threads, even for the same object ID. This means that each of our servants may be processing requests for several threads at the same time.

In the remove operation, not only must we coordinate access to shared object state with the creation operations, but also we must ensure that the object's servant is not destroyed until all requests it is handling have completed. Otherwise, other request invocations might try to access a data member of the deleted servant, and potentially that could cause the server application to crash. We use servant reference counting to solve this issue.

21.6.4 Controller_impl Servant Class

The thermometer and thermostat creation functions supplied by the Controller and the Thermometer::remove method all need to access the set of asset numbers kept by the Controller_impl, the servant for the Controller object. To coordinate access to this set of asset numbers, we introduce the necessary locking variables in a place that is accessible to both servant types. Following is the revised

Controller_impl class.

#include <set> #include <string>

#include <ace/Synch_T.h> #include "CCSS.hh"

class Controller_impl : public virtual POA_CCS::Controller { public:

Controller_impl(

// CORBA operations.

virtual CCS::Controller::ThermometerSeq *

list() throw(CORBA::SystemException);

virtual void

find(CCS::Controller::SearchSeq & slist) throw(CORBA::SystemException);

virtual void change(

const CCS::Controller::Thermostat Seq & tlist, CORBA::Short delta

) throw( CORBA::SystemException, CCS::Controller::EChange

);

// Thermometer and Thermostat creation functions. virtual CCS::Thermometer_ptr

842

IT-SC book: Advanced CORBA® Programming with C++

create_thermometer( CCS::AssetType asset_num, const char * location

) throw( CORBA::SystemException,

CCS::Controller::DuplicateAsset

);

virtual CCS::Thermostat_ptr create_thermostat(

CCS::AssetType asset_num, const char * location, CCS::TempType initial_temp

) throw( CORBA::SystemException,

CCS::Controller::DuplicateAsset,

CCS::Thermostat::BadTemp

);

//Public mutex for modifying assets. ACE_Mutex m_assets_mutex;

//Helper functions to allow thermometers and

//thermostats to add themselves to the m_assets set,

//to remove themselves again, and to check for

//existence. These functions assume that the caller

//acquires the m_assets_mutex first.

m_assets.insert(anum);

}

void remove_impl(CCS::AssetType anum)

{

m_assets.erase(anum);

}

CORBA::Boolean exists(CCS::AssetType anum)

{

return m_assets.find(anum) != m_assets.end();

// copy not supported Controller_impl(const Controller_impl &); void operator=(const Controller_impl &);

// Helper class for find() operation not shown.

};

843

IT-SC book: Advanced CORBA® Programming with C++

We add a public data member called m_assets_mutex to this class to protect the m_assets set. The public helper functions add_impl, remove_impl, and exists provide access to this set, but they assume that the caller locks the m_assets_mutex first.

21.6.5 Implementing Creation Operations

The Controller_impl::create_thermometer method is the same as originally shown in Section 12.3.2 except for the code required for thread synchronization. Because the creation of thermometers and thermostats both require the same multithreading synchronization, we show only the revised create_thermometer method.

CCS::Thermometer_ptr Controller_impl::

create_thermometer(CCS::AssetType anum, const char * loc) throw(CORBA::SystemException, CCS::Controller::DuplicateAsset)

{

//Open a nested scope to limit the extent of

//the guard object.

{

// Lock the mutex.

ACE_Guard<ACE_Mutex> guard(m_assets_mutex);

//Make sure the asset number is new. if (exists(anum))

throw CCS::Controller::DuplicateAsset();

//Add the device to the network, program its location,

//and add it to the m_assets map.

if (ICP_online(anum) != 0) abort();

if (ICP_set(anum, "location", loc) != 0) abort()

add_impl(anum);

}

// Create a reference for the new thermometer. return make_dref(m_poa, anum);

}

After locking the m_assets_mutex, we perform creation in exactly the same way as the original code shown in Section 12.3.2. Note that because we throw exceptions if necessary—such as a CCS::Controller::DuplicateAsset exception if we find that the device already exists—our use of the ACE_Guard<ACE_Mutex> to unlock the m_assets_mutex in its destructor is very helpful. It makes it impossible to forget to unlock the mutex even if an exception occurs.

As in the original version of this function, we next mark the device as being on-line and set its location. After that we use the make_dref helper function from Section

844

IT-SC book: Advanced CORBA® Programming with C++

12.3.2 to create the new Thermometer object and its reference. Finally, we return the object reference for the new Thermometer object.

Note that we use a nested scope to control the lifetime of the

ACE_Guard<ACE_Mutex> lock so that it unlocks the m_assets_mutex before the invocation of make_dref. If we were to invoke make_dref with the mutex locked, there is a potential for deadlock. The definition of make_dref shows why.

static CCS::Thermometer_ptr make_dref(PortableServer::POA_ptr poa, CCS::AssetType anum)

{

//Convert asset number to OID. ostrstream ostr;

ostr < anum < ends;

char * anum_str = ostr.str(); PortableServer::ObjectId_var oid

=PortableServer::string_to_ObjectId(anum_str); ostr.rdbuf()->freeze(0);

//Look at the model via the network to determine

//the repository ID.

char buf[32];

if (ICP_get(anum, "model", buf, sizeof(buf)) != 0) abort();

const char * rep_id = strcmp(buf, "Sens-A-Temp") == 0

? "IDL:acme.com/CCS/Thermometer:1.0" : "IDL:acme.com/CCS/Thermostat:1.0";:

"IDL:acme.com/CCS/Thermostat:1.0"; // Make a new reference. CORBA::Object_var obj

= poa->create_reference_with_id(oid, rep_id); return CCS::Thermometer::_narrow(obj);

}

Because make_dref narrows the object reference it creates, it could cause the ORB to invoke the is_a operation on the new object. The object would not yet have a servant at that point to service the is_a request, so the POA hosting the object would call the servant locator to provide a servant. As Section 21.6.7 shows, the servant locator implementation attempts to lock the same mutex in its preinvoke function.

There are two reasons why it is safe to invoke make_dref under these circumstances without holding the lock on m_assets_mutex.

You destroy a device by invoking its remove operation. However, you cannot invoke remove (or any other operation) on the device until you have its object reference. Because make_dref, which at this point has not yet been invoked for the new device, creates the object reference, the object reference is not available to any client. Device removal is not possible until after create_thermometer returns.

845

IT-SC book: Advanced CORBA® Programming with C++

Because create_thermometer acquires the mutex lock before it adds the asset number of the device to the Controller_impl's set of known asset numbers, any other thread trying to create the same device (or any other device) is blocked. After the mutex is unlocked, the waiting thread will lock the mutex, see that the device already exists, and throw the CCS::Controller::DuplicateAsset exception without invoking make_dref.

21.6.6 DeviceLocator_impl Servant Locator

Our DeviceLocator_impl servant locator class does all the work required for servant eviction. It uses the m_assets_mutex to serialize access to the evictor queue and to its own active object map, so it needs no new data members or member functions. Therefore, the following DeviceLocator_impl class definition is identical to the original one shown in Section 12.6.3.

class DeviceLocator_impl :

public virtual POA_PortableServer::ServantLocator { public:

DeviceLocator_impl(Controller_impl * ctrl);

const PortableServer::ObjectId & oid,

) throw(CORBA::SystemException) {}

// Copy not supported

DeviceLocator_impl(const DeviceLocator_impl &);

846

IT-SC book: Advanced CORBA® Programming with C++

void operator=(const DeviceLocator_impl &);

};

21.6.7 Implementing preinvoke

Our implementation of preinvoke must ensure that the target device still exists, create a servant for it if necessary, and possibly evict the LRU servant if the evictor queue is full. This means that it must access the assets set via the controller's exists helper function, and it must modify the evictor queue to add the new servant and possibly to evict another one. It must also store information about the servant in its own active object map.

Because the POA may invoke the preinvoke function simultaneously from multiple threads, we must serialize access to all the shared data structures. One approach would be to create a separate mutex for each one. However, this approach can cause problems if we must acquire two or more of the mutex locks together in different parts of our code. Specifically, if the various parts of our code attempt to acquire the mutex locks in different orders, we could deadlock because of different threads each having acquired a different portion of the group of mutex locks but being blocked by the others from acquiring the rest.

We avoid the potential for deadlock by instead using a single mutex, the m_assets_mutex in the Controller_impl, to protect access to all shared data. Following is the revised implementation of preinvoke that uses this mutex.

PortableServer::Servant DeviceLocator_impl:: preinvoke(

const PortableServer::ObjectId & oid,

) throw(CORBA::SystemException, PortableServer::ForwardRequest)

{

//Convert object id into asset number. CORBA::String_var oid_string;

try {

oid_string = PortableServer::ObjectId_to_string(oid); } catch (const CORBA::BAD_PARAM &) {

throw CORBA::OBJECT_NOT_EXIST();

}

istrstream istr(oid_string.in()); CCS::AssetType anum;

istr >> anum; if (istr.fail())

throw CORBA::OBJECT_NOT_EXIST();

//Acquire the mutex lock.

ACE_Guard<ACE_Mutex> guard(m_ctrl->m_assets_mutex);

847

IT-SC book: Advanced CORBA® Programming with C++

//Check whether the device is known. if (!m_ctrl->exists(anum))

throw CORBA::OBJECT_NOT_EXIST();

//Look at the object map to find out whether

//we have a servant in memory.

Thermometer_impl * servant;

ActiveObjectMap::iterator servant_pos = m_aom.find(anum); if (servant_pos == m_aom.end()) {

//No servant in memory. If evictor queue is full,

//evict servant at head of queue.

if (m_eq.size() == MAX_EQ_SIZE) { servant = m_eq.back(); m_aom.erase(servant->m_anum); m_eq.pop_back(); servant->_remove_ref();

}

//Instantiate correct type of servant. char buf[32];

if (ICP_get(anum, "model", buf, sizeof(buf)) != 0) abort();

if (strcmp(buf, "Sens-A-Temp") == 0) servant = new Thermometer_impl(anum);

else

servant = new Thermostat_impl(anum);

}else {

//Servant already in memory.

//If operation is "remove", also remove entry from

//active object map -- the object is about to be deleted. if (strcmp(operation, "remove") == 0)

m_aom.erase(servant_pos);

}

//We found a servant, or just instantiated it.

//If the operation is not a remove, move

//the servant to the tail of the evictor queue

//and update its queue position in the map. if (strcmp(operation, "remove") != 0) {

m_eq.push_front(servant); m_aom[anum] = m_eq.begin();

} else

m_ctrl->remove_impl(anum); // Mark device as removed. return servant;

}

This implementation is identical to the one in Section 12.6.3 except for the following changes.

We lock the m_assets_mutex before we check for device existence, and we keep it locked until the end of the function to ensure that the evictor queue and active object map do not get corrupted by concurrent access.

848

IT-SC book: Advanced CORBA® Programming with C++

When evicting a servant, we invoke _remove_ref on it instead of directly deleting it. Note that we do not invoke _add_ref on servants that we find already in memory; the only time _remove_ref is invoked on a servant is when we evict it or when the object it incarnates is the target of a remove request. Before dispatching the actual request to the servant (after calling preinvoke and before calling postinvoke), the POA increments the servant's reference count to ensure that the servant remains viable for the duration of the request.

21.6.8 Implementing the Thermometer Servant

Because preinvoke performs the hard work of properly updating our shared data, the Thermometer_impl class is trivial. First, the remove method simply sets the m_removed member variable to mark the fact that the target object has been destroyed. Then it invokes _remove_ref on itself.

void Thermometer_impl::

remove() throw(CORBA::SystemException)

{

m_removed = true; _remove_ref();

}

Note that remove does not need to modify the servant locator's active object map or the evictor queue. By checking the name of the operation in preinvoke, the servant locator can effectively evict any servant processing a remove request. Note also that remove does not remove the target device from the controller's set of assets. For several reasons, the servant locator preinvoke method also takes care of this.

There may be other requests in progress on this servant in other threads. If we made remove responsible for removing the target's asset number from the controller's set, the other threads might see the change and get confused. At that point, they would appear to be running requests for an object that no longer exists.

The preinvoke method must prevent new threads from trying to create a new servant for a removed object. If remove were instead responsible for removing the target's asset number from the controller's set, it would have to reacquire the m_assets_mutex and remove the target's entry. Before it could reacquire the mutex, however, another thread could intervene and invoke preinvoke to get a servant for the same object. The preinvoke method would see that the asset number still exists (because remove had not yet executed) but would find no servant in memory, so it would create a new servant. As soon as remove continued and removed the target's asset number, the new servant that preinvoke created would become a memory leak, at least for awhile. Eventually, if enough requests were to arrive for other objects, the servant would make its way to the head of the evictor queue and would be destroyed. By making preinvoke remove the target's asset number from the controller's set, we can avoid all this. The next time the

849

IT-SC book: Advanced CORBA® Programming with C++

POA invokes preinvoke for the same target object, exists returns false, and preinvoke raises OBJECT_NOT_EXIST.

For similar reasons, remove cannot send an ICP message to mark the device as off-line because requests that are already in progress for the same device must be allowed continued access to that device until they complete. For example, assume that one client invokes the temperature attribute at the same time that another client invokes the remove operation on the same object. Also assume that the temperature request arrives slightly ahead of the remove request, but then its thread gets preempted. The remove method proceeds to mark the device off-line and then completes. When the temperature method starts executing again and tries to send an ICP message to its device, it will fail because the device is no longer on-line.

The only place where we can safely mark the device off-line is in the Thermometer_impl destructor. By the time it is executed, all other threads are guaranteed to have finished using the servant (as long as they all performed proper reference counting of the servant). The destructor executes as soon as the last thread using the servant invokes _remove_ref. The Thermometer_impl destructor looks like this:

Thermometer_impl:: ~Thermometer_impl()

{

if (m_removed && ICP_offline(m_anum) != 0) abort();

}

The destructor checks the m_removed flag instead of unconditionally marking the device as off-line. Servants can also be removed because of eviction, in which case only the servant, and not the object, is being destroyed.

Finally, note that we must derive the Thermometer_impl servant class from

PortableServer::RefCountServantBase (see Section 11.7.5) so that it inherits thread-safe implementations of the _add_ref and _remove_ref reference counting functions.

21.6.9 Evaluating the Multithreaded Evictor

As you can see from our example, implementing the Evictor pattern for a multithreaded application using a servant locator is not much more difficult than implementing it for the single-threaded case. The only changes are as follows.

We add a mutex variable to Controller_impl so that we can protect its set of asset numbers from concurrent access.

850