The Pegasus Meta Dispatcher is a set of classes that extend the existing MessageQueue messaging system to be dynamic, asynchronous, and multithreaded. The primary classes consist of the folowing:
Class | Derived from | Source file |
---|---|---|
cimom | MessageQueue | Pegasus/Common/Cimom.h |
MessageQueueService | MessageQueue | Pegasus/Common/MessageQueueServices.h |
CimomMessage | Message | Pegasus/Common/CimomMessage.h |
AsyncOpNode | n/a | Pegasus/Common/AsyncOpNode.h |
AsyncDQueue | unlocked_dq | Pegasus/Common/DQueue.h |
IPC classes | n/a | Pegasus/Common/IPC.h |
Threading classes | n/a | Pegasus/Common/Thread.h |
The Meta Dispatcher has two primary goals:
Three points are necessary to avoid deadlocks and to provide pluggable services in Pegaus. The first thing is independent execution paths of service modules. i.e., each service must have its own thread(s), which must not intersect with the thread(s) of other services. Intersection of execution paths can occur indirectly through IPC objects such as mutexes, conditions, and semaphores.
The second point that is necessary is interface abstraction, which the Meta Dispatcher provides through C++ polymorphism. This allows pluggable services. i.e., one service can replace another and the system will continue to function (hopefully in an improved manner).
The third point that is neccesary is a central message broker that isolates services from each other, thereby preventing deadlocks. The central message broker also provides message responses for services that are paused, stopped, or not present (plugged in).
The Meta Dispatcher therefore acts as a central message hub. Services communicate with each other via the Meta Dispatcher.
Service A--Message----1----> (block on semaphore) | | Meta Dispatcher| | Message----2->Service B | | (Signal Semaphore) <---Response---3-- + | Service A <--Response--4---+The numbered steps above are as follows:
AsyncMessage
and
AsyncOpNode
and sends that message to Service
B by calling
MessageQueueService::SendWait
. The calling thread
blocks on the client emaphore until the response is ready.
The concepts explained below are all contained in the test
program for the Meta Dispatcher, which is located in
$PEGASUS_HOME/src/Pegasus/Common/tests/MessageQueueService/
Services (classes derived from
MessageQueueService
must register their presence
with the Meta Dispatcher. This is done as follows (taken from
the test program):
// Define our service class class MessageQueueClient : public MessageQueueService { public: typedef MessageQueueService Base; MessageQueueClient(char *name) : Base(name, MessageQueue::getNextQueueId(), 0, message_mask::type_cimom | message_mask::type_service | message_mask::ha_request | message_mask::ha_reply | message_mask::ha_async ), client_xid(1) { _client_capabilities = Base::_capabilities; _client_mask = Base::_mask; } virtual ~MessageQueueClient(void) { } // method to indicate acceptance of message to // Meta Dispatcher virtual Boolean messageOK(const Message *msg); // function to send a request to another service void send_test_request(char *greeting, Uint32 qid); Uint32 get_qid(void); Uint32 _client_capabilities; Uint32 _client_mask; // method to receive messages from the Meta Dispatcher, // MUST be defined virtual void _handle_async_request(AsyncRequest *req); AtomicInt client_xid; };
// Create our Service MessageQueueClient *q_client = new MessageQueueClient("test client");
// Register our service with the Meta Dispatcher q_client->register_service("test client", q_client->_client_capabilities, q_client->_client_mask); cout << " client registered " << endl;
The MessageQueueService class has an api for finding other
services. This api is built using messages that are defined in
CimomMessage.h
. Here is an example from the test
program:
ArrayThe code sample above shows how to find services by their name. The api also allows finding services by their capabilities or the messages they support. Note that the return is an array of Queue IDs. It is possible, for example, to find multiple services.services; while( services.size() == 0 ) { q_client->find_services(String("test server"), 0, 0, &services); pegasus_yield(); } cout << "found server at " << services[0] << endl;
The "handle" for a services is its Queue ID. Once you have the Queue ID you can send a message to that service. The example above shows one way to get a service's Queue ID. Here is an example that shows how to send that service a message.
class test_request : public AsyncRequest { public: typedef AsyncRequest Base; test_request(Uint32 routing, AsyncOpNode *op, Uint32 destination, Uint32 response, char *message) : Base(0x04100000, Message::getNextKey(), routing, 0, op, destination, response, true), greeting(message) { } virtual ~test_request(void) { } String greeting; }; class test_response : public AsyncReply { public: typedef AsyncReply Base; test_response(Uint32 key, Uint32 routing, AsyncOpNode *op, Uint32 result, Uint32 destination, char *message) : Base(0x04200000, key, routing, 0, op, result, destination, true), greeting(message) { } virtual ~test_response(void) { } String greeting; };The function
send_test_request
shows everything
that is necessary to send a message to another service and
process the reply.
void MessageQueueClient::send_test_request(char *greeting, Uint32 qid) {
test_request *req = new test_request(Base::get_next_xid(), 0, qid, // destination queue ID _queueId, // my own queue ID greeting); // message parameter
MessageQueueService::SendWait
AsyncMessage *response = SendWait(req);
if( response != 0 ) { msg_count++; delete response; cout << " test message " << msg_count.value() << endl; } delete req; }
SendWait
interface creates and disposes of
everything else.To handle messages the service needs to implement the following methods.
virtual Boolean MessageOK(const Message
*)
Boolean MessageQueueServer::messageOK(const Message *msg) { if(msg->getMask() & message_mask::ha_async) { if( msg->getType() == 0x04100000 || msg->getType() == async_messages::CIMSERVICE_STOP || msg->getType() == async_messages::CIMSERVICE_PAUSE || msg->getType() == async_messages::CIMSERVICE_RESUME ) return true; } return false; }
virtual Boolean accept_async(AsyncOpNode
*operation)
(optional) virtual void _handle_incoming_operation(AsyncOpNode
*)
void MessageQueueServer::_handle_incoming_operation(AsyncOpNode *op) { if ( operation != 0 ) { Message *rq = operation->get_request(); PEGASUS_ASSERT(rq != 0 ); PEGASUS_ASSERT(rq->getMask() & message_mask::ha_async ); PEGASUS_ASSERT(rq->getMask() & message_mask::ha_request); _handle_async_request(static_cast<AsyncRequest *>(rq)); } return; }
virtual void _handle_async_request(AsyncRequest *)
Base::_handle_async_request(req)
void MessageQueueServer::_handle_async_request(AsyncRequest *req) { if (req->getType() == 0x04100000 ) { req->op->processing(); handle_test_request(req); // Message Handler } else if ( req->getType() == async_messages::CIMSERVICE_STOP ) { req->op->processing(); handle_CimServiceStop(static_cast(req)); } else Base::_handle_async_request(req); // Give it to the Base !! }
if( msg->getType() == 0x04100000 ) {
test_response *resp = new test_response(msg->getKey(), msg->getRouting(), msg->op, async_results::OK, msg->dest, "i am a test response");
_completeAsyncResponse(msg, resp, ASYNC_OPSTATE_COMPLETE, 0); }
class PEGASUS_COMMON_LINKAGE cimom : public MessageQueue { public : cimom(void); virtual ~cimom(void) ; Boolean moduleChange(struct timeval last); Uint32 getModuleCount(void); Uint32 getModuleIDs(Uint32 *ids, Uint32 count) throw(IPCException); AsyncOpNode *get_cached_op(void) throw(IPCException); void cache_op(AsyncOpNode *op) throw(IPCException); void set_default_op_timeout(const struct timeval *buffer); void get_default_op_timeout(struct timeval *timeout) const ; virtual void handleEnqueue(); void register_module(RegisterCimService *msg); void deregister_module(Uint32 quid); void update_module(UpdateCimService *msg ); void ioctl(AsyncIoctl *msg ); void find_service_q(FindServiceQueue *msg ); void enumerate_service(EnumerateService *msg ); Boolean route_async(AsyncOpNode *operation); void _shutdown_routed_queue(void); protected: Uint32 get_module_q(const String & name); void _make_response(AsyncRequest *req, Uint32 code); void _completeAsyncResponse(AsyncRequest *request, AsyncReply *reply, Uint32 state, Uint32 flag); private: struct timeval _default_op_timeout; struct timeval _last_module_change; DQueue<message_module> _modules; DQueue<AsyncOpNode> _recycle; AsyncDQueue<AsyncOpNode> _routed_ops; DQueue<AsyncOpNode> _internal_ops; static PEGASUS_THREAD_RETURN PEGASUS_THREAD_CDECL _routing_proc(void *); Thread _routing_thread; static Uint32 get_xid(void); void _handle_cimom_op(AsyncOpNode *op, Thread *thread, MessageQueue *queue); Uint32 _ioctl(Uint32, Uint32, void *); AtomicInt _die; AtomicInt _routed_queue_shutdown; static AtomicInt _xid; // CIMOperationRequestDispatcher *_cim_dispatcher; // CIMOperationResponseEncoder *_cim_encoder; // CIMOperationRequestDecoder *_cim_decoder; // CIMRepository *_repository; };
class message_module; class PEGASUS_COMMON_LINKAGE MessageQueueService : public MessageQueue { public: typedef MessageQueue Base; MessageQueueService(const char *name, Uint32 queueID, Uint32 capabilities, Uint32 mask) ; virtual ~MessageQueueService(void); virtual void handle_heartbeat_request(AsyncRequest *req); virtual void handle_heartbeat_reply(AsyncReply *rep); virtual void handle_AsyncIoctl(AsyncIoctl *req); virtual void handle_CimServiceStart(CimServiceStart *req); virtual void handle_CimServiceStop(CimServiceStop *req); virtual void handle_CimServicePause(CimServicePause *req); virtual void handle_CimServiceResume(CimServiceResume *req); virtual void handle_AsyncOperationStart(AsyncOperationStart *req); virtual void handle_AsyncOperationResult(AsyncOperationResult *req); virtual Boolean accept_async(AsyncOpNode *op); virtual Boolean messageOK(const Message *msg) ; AsyncReply *SendWait(AsyncRequest *request); void _completeAsyncResponse(AsyncRequest *request, AsyncReply *reply, Uint32 state, Uint32 flag); Boolean register_service(String name, Uint32 capabilities, Uint32 mask); Boolean update_service(Uint32 capabilities, Uint32 mask); Boolean deregister_service(void); virtual void _shutdown_incoming_queue(void); void find_services(String name, Uint32 capabilities, Uint32 mask, Array<Uint32> *results); void enumerate_service(Uint32 queue, message_module *result); Uint32 get_next_xid(void); AsyncOpNode *get_op(void); void return_op(AsyncOpNode *op); Uint32 _capabilities; Uint32 _mask; AtomicInt _die; protected: virtual void _handle_incoming_operation(AsyncOpNode *operation, Thread *thread, MessageQueue *queue); virtual void _handle_async_request(AsyncRequest *req); virtual void _make_response(AsyncRequest *req, Uint32 code); cimom *_meta_dispatcher; private: void handleEnqueue(); DQueue<AsyncOpNode> _pending; AsyncDQueue<AsyncOpNode> _incoming; static PEGASUS_THREAD_RETURN PEGASUS_THREAD_CDECL _req_proc(void *); AtomicInt _incoming_queue_shutdown; Thread _req_thread; struct timeval _default_op_timeout; static AtomicInt _xid; };
extern const Uint32 CIMOM_Q_ID; class AsyncOpNode; class PEGASUS_COMMON_LINKAGE async_results { public: static const Uint32 OK; static const Uint32 PARAMETER_ERROR; static const Uint32 MODULE_ALREADY_REGISTERED; static const Uint32 MODULE_NOT_FOUND; static const Uint32 INTERNAL_ERROR; static const Uint32 ASYNC_STARTED; static const Uint32 ASYNC_PROCESSING; static const Uint32 ASYNC_COMPLETE; static const Uint32 ASYNC_CANCELLED; static const Uint32 ASYNC_PAUSED; static const Uint32 ASYNC_RESUMED; static const Uint32 CIM_SERVICE_STARTED; static const Uint32 CIM_SERVICE_STOPPED; static const Uint32 CIM_SERVICE_PAUSED; static const Uint32 CIM_SERVICE_RESUMED; static const Uint32 CIM_NAK; static const Uint32 ASYNC_PHASE_COMPLETE; static const Uint32 ASYNC_CHILD_COMPLETE; static const Uint32 ASYNC_PHASE_STARTED; static const Uint32 ASYNC_CHILD_STARTED; static const Uint32 CIM_PAUSED; static const Uint32 CIM_STOPPED; }; class PEGASUS_COMMON_LINKAGE async_messages { public: static const Uint32 HEARTBEAT; static const Uint32 REPLY; static const Uint32 REGISTER_CIM_SERVICE; static const Uint32 DEREGISTER_CIM_SERVICE; static const Uint32 UPDATE_CIM_SERVICE; static const Uint32 IOCTL; static const Uint32 CIMSERVICE_START; static const Uint32 CIMSERVICE_STOP; static const Uint32 CIMSERVICE_PAUSE; static const Uint32 CIMSERVICE_RESUME; static const Uint32 ASYNC_OP_START; static const Uint32 ASYNC_OP_RESULT; static const Uint32 ASYNC_LEGACY_OP_START; static const Uint32 ASYNC_LEGACY_OP_RESULT; static const Uint32 FIND_SERVICE_Q; static const Uint32 FIND_SERVICE_Q_RESULT; static const Uint32 ENUMERATE_SERVICE; static const Uint32 ENUMERATE_SERVICE_RESULT; }; class PEGASUS_COMMON_LINKAGE AsyncMessage : public Message { public: AsyncMessage(Uint32 type, Uint32 key, Uint32 routing, Uint32 mask, AsyncOpNode *operation); virtual ~AsyncMessage(void) { } Boolean operator ==(void *key); Boolean operator ==(const AsyncMessage& msg); AsyncOpNode *op; Thread *_myself; MessageQueue *_service; }; inline Boolean AsyncMessage::operator ==(void *key) { if( key == reinterpret_cast<void *>(this)) return true; return false; } inline Boolean AsyncMessage::operator ==(const AsyncMessage& msg) { return this->operator==(reinterpret_cast<void *>(const_cast<AsyncMessage *>(&msg))); } class PEGASUS_COMMON_LINKAGE AsyncRequest : public AsyncMessage { public: AsyncRequest(Uint32 type, Uint32 key, Uint32 routing, Uint32 mask, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking); virtual ~AsyncRequest(void) { } Uint32 dest; Uint32 resp; Boolean block; }; class PEGASUS_COMMON_LINKAGE AsyncReply : public AsyncMessage { public: AsyncReply(Uint32 type, Uint32 key, Uint32 routing, Uint32 mask, AsyncOpNode *operation, Uint32 result_code, Uint32 destination, Boolean blocking); virtual ~AsyncReply(void) { if(op != 0 ) delete op; } Uint32 result; Uint32 dest; Boolean block; }; class PEGASUS_COMMON_LINKAGE RegisterCimService : public AsyncRequest { public: RegisterCimService(Uint32 routing, AsyncOpNode *operation, Boolean blocking, String service_name, Uint32 service_capabilities, Uint32 service_mask, Uint32 service_queue); virtual ~RegisterCimService(void) { } String name; Uint32 capabilities; Uint32 mask; Uint32 queue; }; class PEGASUS_COMMON_LINKAGE DeRegisterCimService : public AsyncRequest { public: DeRegisterCimService(Uint32 routing, AsyncOpNode *operation, Boolean blocking, Uint32 service_queue); virtual ~DeRegisterCimService(void) { } Uint32 queue; } ; class PEGASUS_COMMON_LINKAGE UpdateCimService : public AsyncRequest { public: UpdateCimService(Uint32 routing, AsyncOpNode *operation, Boolean blocking, Uint32 service_queue, Uint32 service_capabilities, Uint32 service_mask); virtual ~UpdateCimService(void) { } Uint32 queue; Uint32 capabilities; Uint32 mask; }; class PEGASUS_COMMON_LINKAGE AsyncIoctl : public AsyncRequest { public: AsyncIoctl(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking, Uint32 code, Uint32 int_param, void *p_param); virtual ~AsyncIoctl(void) { } enum { IO_CLOSE, IO_OPEN, IO_SOURCE_QUENCH, IO_SERVICE_DEFINED }; Uint32 ctl; Uint32 intp; void *voidp; }; class PEGASUS_COMMON_LINKAGE CimServiceStart : public AsyncRequest { public: CimServiceStart(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking); virtual ~CimServiceStart(void) { } }; class PEGASUS_COMMON_LINKAGE CimServiceStop : public AsyncRequest { public: CimServiceStop(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking); virtual ~CimServiceStop(void) { } }; class PEGASUS_COMMON_LINKAGE CimServicePause : public AsyncRequest { public: CimServicePause(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking); virtual ~CimServicePause(void) { } }; class PEGASUS_COMMON_LINKAGE CimServiceResume : public AsyncRequest { public: CimServiceResume(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking); virtual ~CimServiceResume(void) { } }; class PEGASUS_COMMON_LINKAGE AsyncOperationStart : public AsyncRequest { public: AsyncOperationStart(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Uint32 response, Boolean blocking, Message *action); virtual ~AsyncOperationStart(void) { } Message *act; }; class PEGASUS_COMMON_LINKAGE AsyncOperationResult : public AsyncReply { public: AsyncOperationResult(Uint32 key, Uint32 routing, AsyncOpNode *operation, Uint32 result_code, Uint32 destination, Uint32 blocking); virtual ~AsyncOperationResult(void) { } }; class PEGASUS_COMMON_LINKAGE AsyncLegacyOperationStart : public AsyncRequest { public: AsyncLegacyOperationStart(Uint32 routing, AsyncOpNode *operation, Uint32 destination, Message *action); virtual ~AsyncLegacyOperationStart(void) { } Message *act; }; class PEGASUS_COMMON_LINKAGE AsyncLegacyOperationResult : public AsyncReply { public: AsyncLegacyOperationResult(Uint32 key, Uint32 routing, AsyncOpNode *operation, Message *result); virtual ~AsyncLegacyOperationResult(void) { } Message *res; }; class PEGASUS_COMMON_LINKAGE FindServiceQueue : public AsyncRequest { public: FindServiceQueue(Uint32 routing, AsyncOpNode *operation, Uint32 response, Boolean blocking, String service_name, Uint32 service_capabilities, Uint32 service_mask); virtual ~FindServiceQueue(void) { } String name; Uint32 capabilities; Uint32 mask; } ; class PEGASUS_COMMON_LINKAGE FindServiceQueueResult : public AsyncReply { public: FindServiceQueueResult(Uint32 key, Uint32 routing, AsyncOpNode *operation, Uint32 result_code, Uint32 destination, Boolean blocking, Array<Uint32> queue_ids); virtual ~FindServiceQueueResult(void) { } Array<Uint32> qids; } ; class PEGASUS_COMMON_LINKAGE EnumerateService : public AsyncRequest { public: EnumerateService(Uint32 routing, AsyncOpNode *operation, Uint32 response, Boolean blocking, Uint32 queue_id); virtual ~EnumerateService(void) { } Uint32 qid; }; class PEGASUS_COMMON_LINKAGE EnumerateServiceResponse : public AsyncReply { public: EnumerateServiceResponse(Uint32 key, Uint32 routing, AsyncOpNode *operation, Uint32 result_code, Uint32 response, Boolean blocking, String service_name, Uint32 service_capabilities, Uint32 service_mask, Uint32 service_qid); virtual ~EnumerateServiceResponse(void) { } String name; Uint32 capabilities; Uint32 mask; Uint32 qid; };
#define ASYNC_OPFLAGS_UNKNOWN 0x00000000 #define ASYNC_OPFLAGS_INTERVAL_REPEAT 0x00000010 #define ASYNC_OPFLAGS_INDICATION 0x00000020 #define ASYNC_OPFLAGS_REMOTE 0x00000040 #define ASYNC_OPFLAGS_LOCAL_OUT_OF_PROC 0x00000080 #define ASYNC_OPFLAGS_PHASED 0x00000001 #define ASYNC_OPFLAGS_PARTIAL 0x00000002 #define ASYNC_OPFLAGS_NORMAL 0x00000000 #define ASYNC_OPFLAGS_SINGLE 0x00000008 #define ASYNC_OPFLAGS_MULTIPLE 0x00000010 #define ASYNC_OPFLAGS_TOTAL 0x00000020 #define ASYNC_OPFLAGS_META_DISPATCHER 0x00000040 #define ASYNC_OPSTATE_UNKNOWN 0x00000000 #define ASYNC_OPSTATE_OFFERED 0x00000001 #define ASYNC_OPSTATE_DECLINED 0x00000002 #define ASYNC_OPSTATE_STARTED 0x00000004 #define ASYNC_OPSTATE_PROCESSING 0x00000008 #define ASYNC_OPSTATE_DELIVER 0x00000010 #define ASYNC_OPSTATE_RESERVE 0x00000020 #define ASYNC_OPSTATE_COMPLETE 0x00000040 #define ASYNC_OPSTATE_TIMEOUT 0x00000080 #define ASYNC_OPSTATE_CANCELLED 0x00000100 #define ASYNC_OPSTATE_PAUSED 0x00000200 #define ASYNC_OPSTATE_SUSPENDED 0x00000400 #define ASYNC_OPSTATE_RESUMED 0x00000800 #define ASYNC_OPSTATE_ORPHANED 0x00001000 #define ASYNC_OPSTATE_RELEASED 0x00002000 class Cimom; class PEGASUS_COMMON_LINKAGE AsyncOpNode { public: AsyncOpNode(void); ~AsyncOpNode(void); Boolean operator == (const void *key) const; Boolean operator == (const AsyncOpNode & node) const; void get_timeout_interval(struct timeval *buffer) ; void set_timeout_interval(const struct timeval *interval); Boolean timeout(void) ; OperationContext & get_context(void) ; void put_request(const Message *request) ; Message *get_request(void) ; void put_response(const Message *response) ; Message *get_response(void) ; Uint32 read_state(void) ; void write_state(Uint32) ; Uint32 read_flags(void); void write_flags(Uint32); void lock(void) throw(IPCException); void unlock(void) throw(IPCException); void udpate(void) throw(IPCException); void deliver(const Uint32 count) throw(IPCException); void reserve(const Uint32 size) throw(IPCException); void processing(void) throw(IPCException) ; void processing(OperationContext *context) throw(IPCException); void complete(void) throw(IPCException) ; void complete(OperationContext *context) throw(IPCException); void release(void); void wait(void); private: Semaphore _client_sem; Mutex _mut; unlocked_dq<Message> _request; unlocked_dq<Message> _response; OperationContext _operation_list; Uint32 _state; Uint32 _flags; Uint32 _offered_count; Uint32 _total_ops; Uint32 _completed_ops; Uint32 _user_data; struct timeval _start; struct timeval _lifetime; struct timeval _updated; struct timeval _timeout_interval; AsyncOpNode *_parent; unlocked_dq<AsyncOpNode> _children; void _reset(unlocked_dq<AsyncOpNode> *dst_q); // the lifetime member is for cache management by the cimom void _set_lifetime(struct timeval *lifetime) ; Boolean _check_lifetime(void) ; Boolean _is_child(void) ; Uint32 _is_parent(void) ; Boolean _is_my_child(const AsyncOpNode & caller) const; void _make_orphan( AsyncOpNode & parent) ; void _adopt_child(AsyncOpNode *child) ; void _disown_child(AsyncOpNode *child) ; friend class cimom; friend class MessageQueueService; };