1 karl 1.3 Using the CIM/XML Pull Operations and FQL query Language
|
2 karl 1.2
3 STATUS
4
|
5 karl 1.3 The Pull operations and FQL query language are incorporated into
6 OpenPegasus 2.14. This readme defines the characteristics, limitations,
7 etc. for that implementation.
|
8 karl 1.2
9 ===========================================
10 OVERVIEW:
11
12 The operation extensions for pull operations defined in the DMTF specification
|
13 karl 1.3 DSP0200 V 1.4 were implemented in Pegasus effective Pegasus version 2.14
|
14 karl 1.2 including Client and Server.
15
16 These operations extend the CIM/XML individual operations to operation
17 sequences where the server must maintain state between operations in a
18 sequence and the client must execute multiple operations to get the full
19 set of instances or instance paths.
20
21 The following new CIM/XML operations as defined in DSP0200 are included;
22
23 -OpenEnumerateInstances
|
24 karl 1.3 -OpenEnumerateInstancePaths
|
25 karl 1.2 -OpenReferenceInstances
26 -OpenReferenceInstancePaths
27 -OpenAssociatiorInstances
28 -OpenAssociatorInstancePaths
29 -OpenQueryInstances
30 -PullInstancesWithPath
31 -PullInstancePaths
32 -PullInstances
33 -CloseEnumeration
|
34 karl 1.3 -EnumerationCount (deprecated by DMTF and incomplete)
|
35 karl 1.2
36 Since the concept of sequences of operations linked together (open, pull, close)
37 is a major extension to the original CIM/XML operation concept of completely
38 independent operations several new pieces of functionality are implemented
|
39 karl 1.3 to control interOperation timeouts, counts of objects to be returned, etc.
|
40 karl 1.2
|
41 karl 1.3 NOTE: More detailed information on the pull operations is contained in:
42 1. The OpenPegasus wiki (pull operations work group)
43 2. The pull operations PEP (preliminary today)
44 3. Presentations in the OpenPegasus wiki, in particular in the
45 pull operations workgroup documentation at:
|
46 karl 1.2
|
47 karl 1.3 https://wiki.opengroup.org/pegasus-wiki/doku.php?id=dev:workgroups:pulloperationsupportworkgroup:pull_operation_support_work_group
48
49
50 OPENPEGASUS CLIENT
|
51 karl 1.2
52 The new operations follow the same pattern as the APIs for existing operations
53 in that:
54
55 1. All errors are handled as CIMException and Exception
56
57 2. The means of inputting parameters are the same except that there are
|
58 karl 1.3 significantly more input parameters with the open operations and for the
59 first time operations return parameters as well as objects in the
60 response. Specifically the open and pull operations return values for
61 enumerationContext which is the identity for a pull sequence and
62 endOfSequence which is the marker the server sends in open and pull
|
63 karl 1.2 responses when it has no more objects to send.
64
|
65 karl 1.3 The significant differences between the open... and original enumerate,
66 associator, and reference operations includes:
|
67 karl 1.2
|
68 karl 1.3 1. The new pull client operations typically require multiple client operations
69 to retrieve a complete set of data (ex. OpenEnumerateInstances and
70 PullInstancesWithPath) are equivalent to the old EnumerateInstances
71 client request.
72
73 2. Processing of parameters on responses (i.e. the endOfSequence and
74 enumerationContext parameters are returned for open and pull operations)
75 These return arguments are used to control the loop retrieving data
76 for an enumeration sequence (ex. the sequence of OpenEnumerateInstances
77 and PullInstancesWithPath that represents a complete enumeration
78 sequence). The sequence continues until the server responds with
79 EndOfSequence is true, and the enumerationContext is the identifier
80 for the sequence.
|
81 karl 1.2
|
82 karl 1.3 3. Numeric arguments (Uint32 and Uint64) include the option of NULL in some
|
83 karl 1.2 cases so they are packaged inside classes Uint32Arg and Uint64Arg in the
|
84 karl 1.3 client API.
85
86 4. The openAsociatorInstances and openReferenceInstances operations ONLY
87 process instances. They do not include the capability to return classes
88 as reference and associator requests do and therefore return CIMInstance
89 rather than CIMObject.
|
90 karl 1.2
|
91 karl 1.3 5. Paths are returned for the instance operations as part of the returned
92 instances (ex. OpenEnumerateInstances and PullInstancesWithPath) where
93 they were not with EnumerateInstances.
|
94 karl 1.2
|
95 karl 1.3 6. The client must maintain state between operations in an enumeration
96 sequence (using the enumerationContext parameter returned with open
97 and pull responses). The client must always return the last
98 enumerationContext received as the server may modify the
99 enumerationContext for each response in an enumeration sequence.
|
100 karl 1.2
|
101 karl 1.3 The client API is defined in the header file:
|
102 karl 1.2
|
103 karl 1.3 pegasus/src/Pegasus/Client/CIMCLient.h
|
104 karl 1.2
105
|
106 karl 1.3 OPENPEGASUS SERVER
|
107 karl 1.2
|
108 karl 1.3 The Pegasus server attempts to always deliver at least some objects
109 in a response. It does not wait to deliver the exact number requested but
110 waits to deliver at least some. The sever will wait about 15 seconds to
111 deliver and if there is nothing delivered from the providers it will then
112 return a response with zero objects in it so that the client does not
113 timeout. This would only occur if the providers are extremely slow (i.e.
114 greater than 15 seconds) in preparing response objects to be delivered.
|
115 karl 1.2
|
116 karl 1.3 The Pegasus server always closes an enumeration sequence upon receipt of any
|
117 karl 1.2 error from the providers, repository, etc. Therefore the server will reject
118 any request that has continueOnError = true;
119
120 Expansion to allow the continue on error may be added in a future version.
121 In any case, the whole purpose of the continue on error is really to allow
122 input from good providers to be mixed with providers that return errors so
|
123 karl 1.3 that generally this would mean simply changing the logic in the return mechanism
|
124 karl 1.2 to not shutdown when an error is received from any given provider.
125
126 Generally we do not believe that the providers need to do much more in the
127 future to support the continueOnError other than possibly allowing the provider
128 to continue processing after it has received an error.
129
|
130 karl 1.3 OPENPEGASUS PROVIDERS
|
131 karl 1.2
|
132 karl 1.3 This implementation (OpenPegasus version 2.14) requires NO changes to the
133 existing providers. The provider APIs operate just as they do with the
134 original operations.
|
135 karl 1.2
136 Because the server processing is different however, there may be some
137 behavior differences primarily because the client now controls the speed of
138 delivery of objects.
139
|
140 karl 1.3 In previous versions of Pegasus, the server attempted to deliver objects as
|
141 karl 1.2 rapidly as then can be put on the network. In the case of HTTP chunked requests
142 they are delivered in chunks of about 100 objects. The primary delay for the
143 providers was the processing of each segment through the server. The server
144 is blocked so that no other segment can proceed through the server until that
145 segment is processed and sent on the network.
|
146 karl 1.3 In the case of non-chunked responses, they are completely gathered in the serve
|
147 karl 1.2 and then delivered as one non-chunked response. There were no delays for the
148 providers, just lots of possible memory use in the server.
149
|
150 karl 1.3 The responses from providers (delivered through the deliver(...) interface) are
|
151 karl 1.2 gathered into segments of about 100 objects and this group of objects is moved
152 through the server to be delivered to the client.
153
|
154 karl 1.3 However with the inclusion of the pull operations, the segments (CIMResponseData
155 objects containing the instance or path objects) from the providers are cached
156 in the server response cache until the next pull request and that number
157 is returned in the response to that pull. Thus, if the client is slow to issue
158 pull requests, the providers might be delayed at some point to reduce memory
159 usage in the server (the delay appears as slow response to the deliver operation).
|
160 karl 1.2
161 In other words, the time to process large sets of responses from the provider
162 now depends on the speed of handling the client.
163
164 It is important to remember in developing providers that the Pegasus server
165 can most efficiently process responses if they are passed from the provider
166 to the server individually or in small arrays of objects rather than the
167 provider gathering very large arrays of objects and sending them to the
168 server.
169
|
170 karl 1.3 FQL (FILTER QUERY LANGUAGE)
171
172 The FQL implementation is complete in accord with the DMTF
173 specification DSP 0212 except for a few issues including:
174
175 1. The regex for the LIKE operation is the same as CQL basic. It provides
176 only the following special characters "." and "*".
177 2. The implementation does not include the comparison of
178 embeddedInstances
179 3. The implementation does not include the handling of Uint8[] as
180 strings.
181
182 For more details on the FQL implementation see the readme.txt in the
183 directory pegasus/src/Pegasus/FQL.
184
185 OpenPegasus will remove these limitations in a future version.
186
187 The FQL implementation includes a large set of sample queries in the
188 directory:
189
190 pegasus/src/Pegasus/FQL/tests/Parser
191 karl 1.3
192 including both good and error generating queries.
193
194 ==================================================================
195 LIMITATIONS IN OPENPEGASUS 2.14
196
197 1. The openQueryInstances does not allow requesting the class on response.
198
199 2. The openEnumerateInstanceNames, openAssociatorNames, and openReferenceNames
200 do not allow use of the query filter. This is because
201 a. The intention is to deprecate these operations completely and remove
202 them in a future version
203 b. They require that the server call the providers with the corresponding
204 enumerate, associators, references to get the full instances to
205 filter
206
207 3. The filterQuery filtering is done by the server, not the providers in this
208 version of OpenPegasus. This will be modified in a future version of
209 OpenPegasus when the provider API extensions for the pull operations
210 have been resolved. Version 2.1 of the CMPI specification will resolve
211 this issue.
212 karl 1.3
213 4. The input parameter continueOnError is processed correctly by the client
214 but the Pegasus server only provides code for 'false' value, since the server
215 does not include logic to continue processing responses after an error is
216 encountered.
217 This is consistent with the statement in the specification that use of
218 this functionality is optional and the fact that the DMTF agrees that all
219 of the issues of continuing after errors have not been clarified.
220
221 5. The operation enumerationCount is not processed by the server today since
222 a) Getting the count would be the same cost as the corresponding
223 enumeration, b) the server does not include a history or estimating
224 mechanism for this to date.
225 NOTE: After a through review as part of the development of the next version
226 of CMPI we have concluded that this operation is probably not worth the
227 effort. Since it is optional, Pegasus will only return the unknown status
228 at this point. Further it is the intention of the DMTF to deprecate this
229 function.
|
230 karl 1.2
|
231 karl 1.3 ===================================================================
232 PULL OPERATION CONFIGURATION PARAMETERS
|
233 karl 1.2
234 The server includes several configuration parameters to set limits on the
|
235 karl 1.3 processing of pull operations.
236
237 RUNTIME CONFIGURATION PARAMETERS
|
238 karl 1.2
|
239 karl 1.3 1. Maximum value of interoperation time (pullOperationsMaxTimeout) -
240 This parameter defines the maximum time allowed between the return of an open
241 or pull response and the receipt of the next pull or a close operation before
242 the server may close the enumeration.
243 The specification allows the server to set a maximum interoperation time and
244 refuse open requests that with requested operationTimeout greater than that time.
245
246 2. Maximum number of objects returned in a single open or pull operation
247 (pullOperationsMaxObjectCount) - The server can set a maximum limit on the
248 number of objects that can be returned in a single open or pull operation
249 with the maxObjectCount parameter. This parameter sets the maximum limit.
250 The absolute maximum allowed without recompiling (and changing a value in
251 Common/Constants.h) is 10000.
252
253 3. Default operationTimeout (pullOperationsdefaultTimeout) - If the client
254 does not specify an operation timeout in the open request, the server uses
255 the value defined by this runtime configuration variable. The default
256 is 30 seconds.
257
258 COMPILE TIME CONFIGURATION PARAMETERS
259
260 karl 1.3 1. Whether the server allows 0 as an interoperation timeout value - The value
261 zero is a special value for the interoperationTimeout in that it tells the
262 server to not timeout the enumeration sequence. This should never be used
263 as it allows a client to open enumeration sequences that will not be
264 cleaned up if the client does not properly terminate them. There is a
265 compile time variable in CIMOperationRequestDispatcher that would allow this
266 to be set so 0 interoperation is allowed but we recommend that it never
267 be used since it removes a significant component of the server management
268 of enumerationContext information.
269
270 With this value for interoperationTimeout, the only way to close an
271 enumeration sequence is to complete all of the pulls or issue the close.
272 If for some reason the sequence is not completed, that enumeration context
273 would remain open indefinitely. Since in Pegasus any open enumeration
274 context uses resources (the context object and any provider responses that
275 have not yet been issued in a response) it would appear that most
276 platforms would not want to allow the existence of enumeration contexts
277 that cannot be closed by the server.
|
278 karl 1.2
|
279 karl 1.3 2. Maximum consecutive pull requests with 0 maxObjectCount - The use of the
|
280 karl 1.2 pull operation with maxObjectCount set to zero could be used to keep an
|
281 karl 1.3 enumeration context open indefinitely (this tells the server to restart the
|
282 karl 1.2 interoperationTimeout but not send any objects in the response). Therefore the
283 specification allows for the server setting maximum limits on this behavior
284 and returning the error CIM_ERR_SERVER_LIMITS_EXCEEDED if this limit is
285 exceeded.
|
286 karl 1.3
|
287 karl 1.2 Note that this is maximum CONSECUTIVE pulls so that issuing a pull with
288 a non-zero count resets this counter.
289
|
290 karl 1.3 Pegasus sets the value of this limit to 1000 and allows the implementer to
291 modify it by compiling with the PEGASUS_MAXIMUM_ZERO_OBJECTCOUNT define in
292 CIMOperationRequestDispatcher.cpp modified.
293
294 3. Time to wait for next response from providers - In the case where providers
295 are responding very slowly, the goal is to generate responses with
296 zero instances at regular intervals to allow the client to keep the
297 enumeration context alive. This wait time is set by a compile time define
298 (PEGASUS_PULL_MAX_OPERATION_WAIT_SEC)in pegasus/src/Pegasus/Common/Constants.h
299 and is set to 15 seconds in the current release.
300
301 4. Time to wait before killing off an enumeration context that is blocked
302 by missing provider responses. In the rare case where providers do not complete
303 their responses to the server there is a compile-time counter that trys to
304 clean up the providers and finally just kill the enumeration sequence after
305 a defined number of consecutive pulls that return zero objects.
306 The limit is defined in pegasus/src/Pegasus/Constants.h
307
308 NOTE: The development team is trying to consolidate all such constants and
309 #define definitions that control overall server characteristics but are not
310 runtime parameters in Constants.h
311 karl 1.3
312 ===================================================================
313 TESTING
314
315 The pull operations are tested primarily with two client programs in the
316 directory Pegasus/Client/tests
317
318 1. pullop and it corresponding Makefile provide extensive tests of the
319 pull operations and comparison of the results with the corresponding non
320 pull operations.
321
322 2. PullErrors tests a number of error scenarios with the pull operations.
323
324 3. cimcli has been extended to allow execution of the pull operations with new
325 operations that parallel the existing operations:
326 enumerateinstances(ei) - corresponding is pullenumerateInstances(pei)
327 ni -> pni enumerateInstanceNames
328 r -> pr references
329 rn -> prn referenceNames
330 a -> pa associators
331 ar -> par associatorNames
|
332 karl 1.2
|
333 karl 1.3 These execute complete pull sequences (open, pull) with cimcli options
334 to control parameters like maxObjectCount, interoperation Timeout, etc.
|
335 karl 1.2
336
|
337 karl 1.3 =============================================================
338 TODO LIST - Post 2.14 release
339 1. Binary operation from OOP. Need to add counter to binary
340 protocol to be able to count objects in response. Generates
341 warnings in things like messageserializer and does not work with
342 OOP right now. Fixed by converting to XML. Concluded that we do not
343 need to do this. The binary response is not really used often
344 in the current environment So double mapping it is not a major issue.
345 Leave this as FUTURE
346 2. Minor TODOs, diagnostics, etc. still in the code. Reduced almost to
347 none now. We are leaving some in as PEGASUS_DEBUG
348 3. Extension to avoid double move of objects in CIMResponseData (one
349 into enumerationContext queue and second to new cimResponseData for
350 response. Want to avoid second move by extending Open/Pull response
351 messages to include count and CIMResponse data to count objects out
352 of queue when converting (avoids the second move). This would mean
353 extending the output writers to allow the count field to be supplied
354 so they would only create output for up to the count supplied.(Make
355 this future beyond bug 9676). This is fairly extensive because it
356 extends beyond CIMResponseData to SCMO and XML writers where the
357 XmlWriters used by encodeXmlResponse would have to have counters
358 karl 1.3 added. Then instead of copying on getCache we would simply pass the
359 cache and count on and the writer would take and remove.
360 4. Add more static tests (currently only OpenEnumerateInstances and
361 OpenEnumerateInstanceNames covered).
362 5. Correct issue between operations and HTTP where we are sending
363 trailers with exceptions. Modify response mechanisms so that we
364 set non-chunked for all responses where we send error responses to
365 avoid the trailers. NOTE: There should be now a bug on this in general
366 where we would want to send an initial error without the trailer. Should
367 have always done that.
368 6. It would be more efficient in dispatcher to always use exception for
369 rejects and change the _reject functions so that they never return
370 when they reject. This is VERY LOW PRIORITY and primarily saves
371 a few lines of code in the reject functions and their calls. Means we
372 would code.
373 _rejectIfEnumerationToBroad(...);
374 _rejectIfThisParameterIncorrect(...);
|
375 karl 1.2
|
376 karl 1.3 instead of
377 if (_rejectIfEnum...)
378 {
379 return true
380 }
381 It would mean that the method trace for the handlers would not return
382 an exit if we rejected. VERY LOW PRIORITY. Possibly FUTURE. No behavior
383 change, just more compact source code but it messes with the method
384 trace logic.
385 7. There are still a couple of template functions around the task of
386 distributing requests to the multiple providers.
387 In fact there are two similar but different templates for
388 the associators(i.e. assoc, ref, etc.) functions and the parallel
389 openAssoc functions. It would be nice to consolidate that logic and
390 further to try to create a non-template form for those functions. LOW
391 PRIORITY
392 8. Trace functions in CIMResponseData,h & cpp should be PEGASUS_DEBUG.
393 Same for trace function in EnumerationContext and
394 EnumerationContextTable
395 9. Question. We added trace in CIMRequestOperationDispatcher.cpp if
396 query enum is recevied with no object path. However, since this is
397 karl 1.3 a provider problem should we be doing something other than a trace
398 (ex. log) since traces are often not really used and they do generate
399 lots of data whereby something like this could get lost. Also, need
400 to review what level of trace if we keep trace.
401 10. Better statistics keeping for open, etc. through cimperf.
402 11. Incorporate a simple test of pull operations into the TestClient
403 12. Define a simpler iteration interface for the client so that
404 the client does not have to directly control the open, pull, etc.
405 operations.
|
406 karl 1.2
|
Return to
readme.pulloperations
CVS log
Up to [Pegasus] / pegasus