1 karl 1.3 Using the CIM/XML Pull Operations and FQL query Language
|
2 karl 1.2
|
3 karl 1.4 Date: 12 November 2014
|
4 karl 1.2
|
5 karl 1.4 STATUS
6
7 The client pull operations and FQL query language are incorporated into
8 OpenPegasus 2.14. consistent and compatible with the DMTF specifications for
9 the pull operations (DMTF DSP0200 v 1.4, and DMTF DSP0212 v 1.01.
10 This readme defines the characteristics, limitations,
|
11 karl 1.3 etc. for that implementation.
|
12 karl 1.2
13 ===========================================
14 OVERVIEW:
15
16 The operation extensions for pull operations defined in the DMTF specification
|
17 karl 1.4 DSP0200 V 1.4 were implemented in OpenPegasus effective version 2.14
|
18 karl 1.2 including Client and Server.
19
20 These operations extend the CIM/XML individual operations to operation
21 sequences where the server must maintain state between operations in a
22 sequence and the client must execute multiple operations to get the full
23 set of instances or instance paths.
24
25 The following new CIM/XML operations as defined in DSP0200 are included;
26
|
27 karl 1.4 -openEnumerateInstances
28 -openEnumerateInstancePaths
29 -openReferenceInstances
30 -openReferenceInstancePaths
31 -openAssociatiorInstances
32 -openAssociatorInstancePaths
33 -openQueryInstances
34 -pullInstancesWithPath
35 -pullInstancePaths
36 -pullInstances
37 -closeEnumeration
38 -enumerationCount (deprecated by DMTF and incomplete)
|
39 karl 1.2
40 Since the concept of sequences of operations linked together (open, pull, close)
41 is a major extension to the original CIM/XML operation concept of completely
42 independent operations several new pieces of functionality are implemented
|
43 karl 1.3 to control interOperation timeouts, counts of objects to be returned, etc.
|
44 karl 1.2
|
45 karl 1.3 NOTE: More detailed information on the pull operations is contained in:
46 1. The OpenPegasus wiki (pull operations work group)
47 2. The pull operations PEP (preliminary today)
48 3. Presentations in the OpenPegasus wiki, in particular in the
49 pull operations workgroup documentation at:
|
50 karl 1.2
|
51 karl 1.3 https://wiki.opengroup.org/pegasus-wiki/doku.php?id=dev:workgroups:pulloperationsupportworkgroup:pull_operation_support_work_group
52
53
54 OPENPEGASUS CLIENT
|
55 karl 1.2
56 The new operations follow the same pattern as the APIs for existing operations
57 in that:
58
|
59 karl 1.4 1. All errors are handled as CIMException and Exception the same as the
60 original client operations. Note, however, that there are additional
61 CIMStatusCodes for the new operations.
|
62 karl 1.2
63 2. The means of inputting parameters are the same except that there are
|
64 karl 1.3 significantly more input parameters with the open operations and for the
65 first time operations return parameters as well as objects in the
66 response. Specifically the open and pull operations return values for
|
67 karl 1.4 the enumerationContext argument which is the identity for a pull sequence and
68 the endOfSequence argument which is the marker the server sends in open and
69 pull responses when it has no more objects to send.
|
70 karl 1.2
|
71 karl 1.3 The significant differences between the open... and original enumerate,
72 associator, and reference operations includes:
|
73 karl 1.2
|
74 karl 1.3 1. The new pull client operations typically require multiple client operations
|
75 karl 1.4 to retrieve a complete set of data (ex. openEnumerateInstances and
76 pullInstancesWithPath) are equivalent to the old enumerateInstances
77 client request. While the whole collection of responses might be returned
78 on the open if the maxObjectCount argument is set larger than the the number
79 of objects in the total response, depending on the speed of delivery of
80 objects from providers, the response might still not deliver everything
81 in the initial response. The client code writer should always assume that
82 multiple requests will be required.
|
83 karl 1.3
84 2. Processing of parameters on responses (i.e. the endOfSequence and
|
85 karl 1.4 enumerationContext arguments are returned for open and pull operations)
|
86 karl 1.3 These return arguments are used to control the loop retrieving data
87 for an enumeration sequence (ex. the sequence of OpenEnumerateInstances
88 and PullInstancesWithPath that represents a complete enumeration
89 sequence). The sequence continues until the server responds with
90 EndOfSequence is true, and the enumerationContext is the identifier
91 for the sequence.
|
92 karl 1.2
|
93 karl 1.3 3. Numeric arguments (Uint32 and Uint64) include the option of NULL in some
|
94 karl 1.2 cases so they are packaged inside classes Uint32Arg and Uint64Arg in the
|
95 karl 1.3 client API.
96
97 4. The openAsociatorInstances and openReferenceInstances operations ONLY
98 process instances. They do not include the capability to return classes
99 as reference and associator requests do and therefore return CIMInstance
100 rather than CIMObject.
|
101 karl 1.2
|
102 karl 1.3 5. Paths are returned for the instance operations as part of the returned
|
103 karl 1.4 instances (ex. openEnumerateInstances and pullInstancesWithPath) where
|
104 karl 1.3 they were not with EnumerateInstances.
|
105 karl 1.2
|
106 karl 1.3 6. The client must maintain state between operations in an enumeration
107 sequence (using the enumerationContext parameter returned with open
108 and pull responses). The client must always return the last
109 enumerationContext received as the server may modify the
110 enumerationContext for each response in an enumeration sequence.
|
111 karl 1.2
|
112 karl 1.3 The client API is defined in the header file:
|
113 karl 1.2
|
114 karl 1.3 pegasus/src/Pegasus/Client/CIMCLient.h
|
115 karl 1.2
116
|
117 karl 1.3 OPENPEGASUS SERVER
|
118 karl 1.2
|
119 karl 1.4 The OpenPegasus server attempts to always deliver at least some objects
|
120 karl 1.3 in a response. It does not wait to deliver the exact number requested but
121 waits to deliver at least some. The sever will wait about 15 seconds to
122 deliver and if there is nothing delivered from the providers it will then
123 return a response with zero objects in it so that the client does not
124 timeout. This would only occur if the providers are extremely slow (i.e.
125 greater than 15 seconds) in preparing response objects to be delivered.
|
126 karl 1.2
|
127 karl 1.4 The OpenPegasus server always closes an enumeration sequence upon receipt of any
|
128 karl 1.2 error from the providers, repository, etc. Therefore the server will reject
|
129 karl 1.4 any request that has the continueOnError argument = true;
|
130 karl 1.2
131 Expansion to allow the continue on error may be added in a future version.
132 In any case, the whole purpose of the continue on error is really to allow
133 input from good providers to be mixed with providers that return errors so
|
134 karl 1.3 that generally this would mean simply changing the logic in the return mechanism
|
135 karl 1.2 to not shutdown when an error is received from any given provider.
136
137 Generally we do not believe that the providers need to do much more in the
138 future to support the continueOnError other than possibly allowing the provider
139 to continue processing after it has received an error.
140
|
141 karl 1.3 OPENPEGASUS PROVIDERS
|
142 karl 1.2
|
143 karl 1.3 This implementation (OpenPegasus version 2.14) requires NO changes to the
144 existing providers. The provider APIs operate just as they do with the
145 original operations.
|
146 karl 1.2
147 Because the server processing is different however, there may be some
148 behavior differences primarily because the client now controls the speed of
149 delivery of objects.
150
|
151 karl 1.3 In previous versions of Pegasus, the server attempted to deliver objects as
|
152 karl 1.2 rapidly as then can be put on the network. In the case of HTTP chunked requests
153 they are delivered in chunks of about 100 objects. The primary delay for the
154 providers was the processing of each segment through the server. The server
155 is blocked so that no other segment can proceed through the server until that
156 segment is processed and sent on the network.
|
157 karl 1.3 In the case of non-chunked responses, they are completely gathered in the serve
|
158 karl 1.2 and then delivered as one non-chunked response. There were no delays for the
159 providers, just lots of possible memory use in the server.
160
|
161 karl 1.3 The responses from providers (delivered through the deliver(...) interface) are
|
162 karl 1.2 gathered into segments of about 100 objects and this group of objects is moved
163 through the server to be delivered to the client.
164
|
165 karl 1.3 However with the inclusion of the pull operations, the segments (CIMResponseData
166 objects containing the instance or path objects) from the providers are cached
167 in the server response cache until the next pull request and that number
168 is returned in the response to that pull. Thus, if the client is slow to issue
169 pull requests, the providers might be delayed at some point to reduce memory
170 usage in the server (the delay appears as slow response to the deliver operation).
|
171 karl 1.2
172 In other words, the time to process large sets of responses from the provider
173 now depends on the speed of handling the client.
174
|
175 karl 1.4 It is important to remember in developing providers that the OpenPegasus server
|
176 karl 1.2 can most efficiently process responses if they are passed from the provider
177 to the server individually or in small arrays of objects rather than the
178 provider gathering very large arrays of objects and sending them to the
179 server.
180
|
181 karl 1.3 FQL (FILTER QUERY LANGUAGE)
182
183 The FQL implementation is complete in accord with the DMTF
184 specification DSP 0212 except for a few issues including:
185
186 1. The regex for the LIKE operation is the same as CQL basic. It provides
187 only the following special characters "." and "*".
188 2. The implementation does not include the comparison of
|
189 karl 1.4 embeddedInstances functionality.
|
190 karl 1.3 3. The implementation does not include the handling of Uint8[] as
191 strings.
|
192 karl 1.4 4. Since the cimserver provides the filtering, it can only filter properties
193 that are returned from the providers. Therefore all of the properties in
194 the filter MUST BE included in any property list provided with the request.
|
195 karl 1.3
196 For more details on the FQL implementation see the readme.txt in the
197 directory pegasus/src/Pegasus/FQL.
198
199 OpenPegasus will remove these limitations in a future version.
200
201 The FQL implementation includes a large set of sample queries in the
202 directory:
203
204 pegasus/src/Pegasus/FQL/tests/Parser
205
206 including both good and error generating queries.
207
208 ==================================================================
209 LIMITATIONS IN OPENPEGASUS 2.14
210
|
211 karl 1.4 1. The openQueryInstances does not allow requesting the class on response
212 (i.e. the returnQueryResultClass argument must be false in this version).
213 Since OpenPegasus is actively proposing that this argument be deprecated
214 we will see what happens in future releases.
|
215 karl 1.3
216 2. The openEnumerateInstanceNames, openAssociatorNames, and openReferenceNames
|
217 karl 1.4 do not allow use of the query filter. This is because:
|
218 karl 1.3 a. The intention is to deprecate these operations completely and remove
|
219 karl 1.4 them in a future versionof both the specificatons and OpenPegasus.
|
220 karl 1.3 b. They require that the server call the providers with the corresponding
221 enumerate, associators, references to get the full instances to
|
222 karl 1.4 filter and then remap this to the corresponding Names operation.
223 We propose that if the user wants just the paths, this can be achieved
224 with the instances operation with the propertylist set empty which
225 requests that the server return no properties.
|
226 karl 1.3
227 3. The filterQuery filtering is done by the server, not the providers in this
228 version of OpenPegasus. This will be modified in a future version of
229 OpenPegasus when the provider API extensions for the pull operations
230 have been resolved. Version 2.1 of the CMPI specification will resolve
|
231 karl 1.4 this issue. However, this has imposed another limitation as mentioned above,
232 the properties that are part of the filter MUST BE included in any
233 propertyList in the request. OpenPegasus does not check to be sure that
234 all properties in the filter are also in the request and would therefore
235 try to filter the response on non-existent properties.
|
236 karl 1.3
237 4. The input parameter continueOnError is processed correctly by the client
238 but the Pegasus server only provides code for 'false' value, since the server
239 does not include logic to continue processing responses after an error is
240 encountered.
241 This is consistent with the statement in the specification that use of
242 this functionality is optional and the fact that the DMTF agrees that all
243 of the issues of continuing after errors have not been clarified.
244
245 5. The operation enumerationCount is not processed by the server today since
246 a) Getting the count would be the same cost as the corresponding
247 enumeration, b) the server does not include a history or estimating
248 mechanism for this to date.
249 NOTE: After a through review as part of the development of the next version
250 of CMPI we have concluded that this operation is probably not worth the
251 effort. Since it is optional, Pegasus will only return the unknown status
252 at this point. Further it is the intention of the DMTF to deprecate this
253 function.
|
254 karl 1.2
|
255 karl 1.3 ===================================================================
256 PULL OPERATION CONFIGURATION PARAMETERS
|
257 karl 1.2
258 The server includes several configuration parameters to set limits on the
|
259 karl 1.3 processing of pull operations.
260
261 RUNTIME CONFIGURATION PARAMETERS
|
262 karl 1.2
|
263 karl 1.3 1. Maximum value of interoperation time (pullOperationsMaxTimeout) -
264 This parameter defines the maximum time allowed between the return of an open
265 or pull response and the receipt of the next pull or a close operation before
266 the server may close the enumeration.
267 The specification allows the server to set a maximum interoperation time and
|
268 karl 1.4 refuse open requests that with requested operationTimeout greater than that
269 time.
|
270 karl 1.3
271 2. Maximum number of objects returned in a single open or pull operation
272 (pullOperationsMaxObjectCount) - The server can set a maximum limit on the
273 number of objects that can be returned in a single open or pull operation
274 with the maxObjectCount parameter. This parameter sets the maximum limit.
275 The absolute maximum allowed without recompiling (and changing a value in
276 Common/Constants.h) is 10000.
277
278 3. Default operationTimeout (pullOperationsdefaultTimeout) - If the client
279 does not specify an operation timeout in the open request, the server uses
280 the value defined by this runtime configuration variable. The default
281 is 30 seconds.
282
|
283 karl 1.4 STATISTICS
284
285 The server does maintain some statistics on the pull operations and outputs
286 these to the console when the server is shutdown. In this version there
287 is no way to output these statistics other than to the console and at
288 shutdown.
289
|
290 karl 1.3 COMPILE TIME CONFIGURATION PARAMETERS
291
292 1. Whether the server allows 0 as an interoperation timeout value - The value
293 zero is a special value for the interoperationTimeout in that it tells the
294 server to not timeout the enumeration sequence. This should never be used
295 as it allows a client to open enumeration sequences that will not be
296 cleaned up if the client does not properly terminate them. There is a
297 compile time variable in CIMOperationRequestDispatcher that would allow this
298 to be set so 0 interoperation is allowed but we recommend that it never
299 be used since it removes a significant component of the server management
300 of enumerationContext information.
301
302 With this value for interoperationTimeout, the only way to close an
303 enumeration sequence is to complete all of the pulls or issue the close.
304 If for some reason the sequence is not completed, that enumeration context
305 would remain open indefinitely. Since in Pegasus any open enumeration
306 context uses resources (the context object and any provider responses that
307 have not yet been issued in a response) it would appear that most
308 platforms would not want to allow the existence of enumeration contexts
309 that cannot be closed by the server.
|
310 karl 1.2
|
311 karl 1.3 2. Maximum consecutive pull requests with 0 maxObjectCount - The use of the
|
312 karl 1.2 pull operation with maxObjectCount set to zero could be used to keep an
|
313 karl 1.3 enumeration context open indefinitely (this tells the server to restart the
|
314 karl 1.2 interoperationTimeout but not send any objects in the response). Therefore the
315 specification allows for the server setting maximum limits on this behavior
316 and returning the error CIM_ERR_SERVER_LIMITS_EXCEEDED if this limit is
317 exceeded.
|
318 karl 1.3
|
319 karl 1.2 Note that this is maximum CONSECUTIVE pulls so that issuing a pull with
320 a non-zero count resets this counter.
321
|
322 karl 1.3 Pegasus sets the value of this limit to 1000 and allows the implementer to
323 modify it by compiling with the PEGASUS_MAXIMUM_ZERO_OBJECTCOUNT define in
324 CIMOperationRequestDispatcher.cpp modified.
325
326 3. Time to wait for next response from providers - In the case where providers
327 are responding very slowly, the goal is to generate responses with
328 zero instances at regular intervals to allow the client to keep the
329 enumeration context alive. This wait time is set by a compile time define
330 (PEGASUS_PULL_MAX_OPERATION_WAIT_SEC)in pegasus/src/Pegasus/Common/Constants.h
331 and is set to 15 seconds in the current release.
332
333 4. Time to wait before killing off an enumeration context that is blocked
334 by missing provider responses. In the rare case where providers do not complete
335 their responses to the server there is a compile-time counter that trys to
336 clean up the providers and finally just kill the enumeration sequence after
337 a defined number of consecutive pulls that return zero objects.
338 The limit is defined in pegasus/src/Pegasus/Constants.h
339
340 NOTE: The development team is trying to consolidate all such constants and
341 #define definitions that control overall server characteristics but are not
342 runtime parameters in Constants.h
343 karl 1.3
344 ===================================================================
|
345 karl 1.4 TESTING PULL OPERATIONS
|
346 karl 1.3
347 The pull operations are tested primarily with two client programs in the
348 directory Pegasus/Client/tests
349
350 1. pullop and it corresponding Makefile provide extensive tests of the
351 pull operations and comparison of the results with the corresponding non
352 pull operations.
353
354 2. PullErrors tests a number of error scenarios with the pull operations.
355
356 3. cimcli has been extended to allow execution of the pull operations with new
357 operations that parallel the existing operations:
358 enumerateinstances(ei) - corresponding is pullenumerateInstances(pei)
359 ni -> pni enumerateInstanceNames
360 r -> pr references
361 rn -> prn referenceNames
362 a -> pa associators
363 ar -> par associatorNames
|
364 karl 1.2
|
365 karl 1.3 These execute complete pull sequences (open, pull) with cimcli options
366 to control parameters like maxObjectCount, interoperation Timeout, etc.
|
367 karl 1.2
368
|
369 karl 1.3 =============================================================
370 TODO LIST - Post 2.14 release
371 1. Binary operation from OOP. Need to add counter to binary
372 protocol to be able to count objects in response. Generates
373 warnings in things like messageserializer and does not work with
374 OOP right now. Fixed by converting to XML. Concluded that we do not
375 need to do this. The binary response is not really used often
376 in the current environment So double mapping it is not a major issue.
377 Leave this as FUTURE
378 2. Minor TODOs, diagnostics, etc. still in the code. Reduced almost to
379 none now. We are leaving some in as PEGASUS_DEBUG
380 3. Extension to avoid double move of objects in CIMResponseData (one
|
381 karl 1.4 into enumerationContext queue and second to new CIMResponseData for
382 response. Want to avoid second move by extending open/pull response
|
383 karl 1.3 messages to include count and CIMResponse data to count objects out
384 of queue when converting (avoids the second move). This would mean
385 extending the output writers to allow the count field to be supplied
386 so they would only create output for up to the count supplied.(Make
387 this future beyond bug 9676). This is fairly extensive because it
388 extends beyond CIMResponseData to SCMO and XML writers where the
389 XmlWriters used by encodeXmlResponse would have to have counters
390 added. Then instead of copying on getCache we would simply pass the
391 cache and count on and the writer would take and remove.
392 4. Add more static tests (currently only OpenEnumerateInstances and
393 OpenEnumerateInstanceNames covered).
394 5. Correct issue between operations and HTTP where we are sending
395 trailers with exceptions. Modify response mechanisms so that we
396 set non-chunked for all responses where we send error responses to
397 avoid the trailers. NOTE: There should be now a bug on this in general
398 where we would want to send an initial error without the trailer. Should
399 have always done that.
400 6. It would be more efficient in dispatcher to always use exception for
401 rejects and change the _reject functions so that they never return
402 when they reject. This is VERY LOW PRIORITY and primarily saves
403 a few lines of code in the reject functions and their calls. Means we
404 karl 1.3 would code.
405 _rejectIfEnumerationToBroad(...);
406 _rejectIfThisParameterIncorrect(...);
|
407 karl 1.2
|
408 karl 1.3 instead of
409 if (_rejectIfEnum...)
410 {
411 return true
412 }
413 It would mean that the method trace for the handlers would not return
414 an exit if we rejected. VERY LOW PRIORITY. Possibly FUTURE. No behavior
415 change, just more compact source code but it messes with the method
416 trace logic.
417 7. There are still a couple of template functions around the task of
418 distributing requests to the multiple providers.
419 In fact there are two similar but different templates for
420 the associators(i.e. assoc, ref, etc.) functions and the parallel
421 openAssoc functions. It would be nice to consolidate that logic and
422 further to try to create a non-template form for those functions. LOW
423 PRIORITY
424 8. Trace functions in CIMResponseData,h & cpp should be PEGASUS_DEBUG.
425 Same for trace function in EnumerationContext and
426 EnumerationContextTable
427 9. Question. We added trace in CIMRequestOperationDispatcher.cpp if
428 query enum is recevied with no object path. However, since this is
429 karl 1.3 a provider problem should we be doing something other than a trace
430 (ex. log) since traces are often not really used and they do generate
431 lots of data whereby something like this could get lost. Also, need
432 to review what level of trace if we keep trace.
433 10. Better statistics keeping for open, etc. through cimperf.
434 11. Incorporate a simple test of pull operations into the TestClient
435 12. Define a simpler iteration interface for the client so that
436 the client does not have to directly control the open, pull, etc.
437 operations.
|
438 karl 1.4 13. Map properties in the FQL Filter to properties in any property list
439 provided.
440 14. Output statistical information on a regular basis rather than just
441 as server shutdown and to the log or somewhere else than just the
442 console.
|
443 karl 1.2
|