pegasus/readme.pulloperations - annotate

Return to readme.pulloperations CVS log

Up to [Pegasus] / pegasus

1 karl 1.1.2.1 Using the CIM/XML Pull Operations 2 3 STATUS 4
5 karl 1.1.2.4 <<< The TODO section is being maintained during the review and checkin process
6 karl 1.1.2.1 to keep track of problems, errors, notes, etc. Must be deleted before 7 checkin to head of tree. Please feel free to add notes, etc in this 8 section as you review/test.>>>>>> 9
10 karl 1.1.2.4 TODO list:
11 karl 1.1.2.3 1. Binary operation from OOP. Need to add counter to binary 12 protocol to be able to count objects in response. Generates 13 warnings in things like messageserializer and does not work with
14 karl 1.1.2.14 OOP right now. Corrected by converting to XML.
15 karl 1.1.2.23 2. OpenExecQuery - Code is incomplete in that it does not include the
16 karl 1.1.2.15 return from the exec query function to the aggregator yet.
17 karl 1.1.2.4 3. Code for Pull part of OpenQueryInstancesRequest a) should be part of 18 the common CIMOperationRequestDispatcher execCommon code. 19 4. The changes to WQLCIMOperationRequestDispatcher and CQL... for handling 20 pull not completed so we feed the responses back to the EnmerationContext 21 queues
22 karl 1.1.2.18 3. Lots of minor TODOs, diagnostics, etc. still in the code
23 karl 1.1.2.14 4. External runtime variables. Proposing that they be fixed for this release
24 karl 1.1.2.15 rather than set by configuration. This should be discussed. Am making
25 karl 1.1.2.16 this a separate bug. See bug 9819 for the changes to cover this.
26 karl 1.1.2.23 5. Consider moving some moreof the code in dispatcher from templates to 27 common functions which would mean adding intermediate classes in 28 CIMMessage but would reduce code size. Currently the change that 29 we should make for this is to create a new message CIMPullRequestData... 30 between CIMRequestDataMessage and the Pull requests since they all 31 have exactly the same input and output pararameters (The 32 CIMResponseData hides the response object differences) 33 6. Extension to avoid double move of objects in CIMResponseData (one
34 karl 1.1.2.3 into enumerationContext queue and second to new cimResponseData for 35 response. Want to avoid second move by extending Open/Pull response 36 messages to include count and CIMResponse data to count objects out
37 karl 1.1.2.23 of queue when converting (avoids the second move). This would mean 38 extending the output writers to allow the count field to be supplied 39 so they would only create output for up to the count supplied. 40 7. Add test for the 0 operationtimeout in PullErrors. 41 8. Add more static tests (currently only OpenEnumerateInstances is covered). 42 9. Could clean up the CIMOperationRequestDecoder.cpp rejectCIMException 43 further since there is some overlap. That is low priority. 44 10.Add exception for to many entries in EnumerationContextTable. The 45 test exists but is ignored right now. 46 7. NEXT TASKS: 47 b. Finish and test the OpenQueryInstances
48 karl 1.1.2.16 c. Clean up TODOs
49 karl 1.1.2.23 d. Find issue that is causing occasional failures under load. It 50 is probably a threading issue between the 3 possible threads 51 that use the EnumerationContext (request thread, provider thread, 52 timeout thread) and a place where not mutuxed. 53 54 21 April 2014 55 1. Reorganizzed code in CIMOperationRequestDecoder.cpp ...Iparam classes. 56 Cleaned up the rejectCIMException code. 57 2. Cleanup some issues in the EnumerationContextTimeout Thread. 58 3. Fixed issue in pullop where it was using zero as default for default 59 operation timeout where it should have been NULL, the DMTF specified 60 default. 61 4. Enabled the test to reject 0 as operation timeout on a request and but 62 controled by a #define in CIMOperationRequestDispatcher.cpp 63 5. Added test for excessive new OperationContext entries in table. 64 65 18 April 2014 66 1. Internationalize remaining messages in CIMOperationRequestDecoder.cpp and 67 clean up the decoders for some of the messages so they use the 68 ...Iparam classes. Note that we did not clean up all of the old message 69 decoders, just the new ones and we did not clean up the Count message 70 karl 1.1.2.23 since it is deprecated. TODO clean these up so the decoder is really 71 consistent. 72 2. Enabled the EnumerationContext Timer and removed code for alternate 73 non-thread solution.
74 karl 1.1.2.16
75 karl 1.1.2.21 17 April 2014 76 1. Reorganized the lock/unlock for handling the EnumerationContext because 77 there were thread windows (i.e. using the context after it was released 78 by other thread). 79 2. Modified code for the InteroperationTimerThread and deleted old code that 80 allowed this whole thread to by bypassed. 81 3. Internationalized a number of CIMOperationRequestDispatcher CIMException 82 response messages.
83 karl 1.1.2.22 4. Fixed stress tests in Client/tests/pullop that were getting timeout. The 84 issue was not the pull code but that fact that the idle client timeout 85 appears to be measureing to the end of the operation and the operation 86 was taking more than 40 seconds which is the default timeout. The idle 87 timeout takes no account of the fact that chunks are being regularly 88 delivered. 89 5. Several minor code changes. 90 NOTE: At this point the code passes extensive tests of the pull operations 91 both in OOP and none OOP mode (but with lots of diagnostics that may 92 be affecting timing).
93 karl 1.1.2.21
94 karl 1.1.2.20 10 April 2014 95 1. Fix issues causing failure with repeated pull operations. 96 2. Consolidate some of the CIMOperationRequestDispatcher.cpp template code 97
98 karl 1.1.2.19 7 April 2104 99 MergeOut of head of tree. 100 101 7 April 2014 102 1. Permanently fixed issue with getting correct number of objects on response. 103 This was introduced when did a temp fix to the OOP processing, in particular 104 removed the condition variable wait in the CIMOperationDispatcher handle 105 functions for opens and pulls. This moved the functionality to kick off 106 open and pull responses to the provider threads if there are not enough 107 objects when the handle is processed. Also added a variable to allow us to 108 test with either a) responses are required to satisfy the original 109 request size or) b, responses are required only to return some objects. 110 We will add statistics to see which of these works best. 111 2. Modified WsmProcessor EnumerationContext class since the name conflicts 112 with the pull Operation EnumerationContext class. 113 3. Created a new intermediate level of CIMMessage, the CIMPullResponseData 114 Message so that all of the open and Pull responses can use common code 115 since there are only two variables (endOfSequence and EnumerationContext) 116 and they are common across all the open and pull responses. 117 4. Removed a number of diagnostics. 118 5. Cleaned up the Dispcatcher so that the open and pulls have common response 119 karl 1.1.2.19 code and that code can be used from the dispatcher handle functions and 120 the provider response functions. 121
122 karl 1.1.2.18 31 March 2014 - Checkin 123 1. Fixed issues in OOP processing of pull operations, in particular 124 issues with cimxml output format when processed through the 125 *InternalXmlEncoder functions. 126 2. Clean up some of the internalXml functionality 127 3. Found issues causing timeout with a particular provider. The issue 128 is that the dispatcher and monitor end up using the same thread so the 129 condition variable in the dispatcher thread stops the monitor. Turned 130 off the conditionVariable in getCache for the moment which means that 131 we get number of responses for open... with 0 objects before the 132 providers can begin to respond. This is only for test. 133 4. Added some statistics for enumerations and display the statistics 134 when we close the server (same as cache statistics) 135 136 12 March 2014 - Mergeout and Mergein 137 1. Mergeout to head of tree for this date and mergein for patch update 138 to bug 9676 139 2. Extensions to pullop tests program and tests. 140 3. Added some diagnostics in looking for OOP issue. 141 4. Removed a number of diagnostics messages and cleaned up code in 142 dispatcher to simplify pull operation processing. 143 karl 1.1.2.18
144 karl 1.1.2.17 15 December 2013 145 1. Mergeout and mergein up to 15 December 2013 146 2. Clean up issues from tests documented in bug 9676 last week. 147 3. Clean up some code in dispatcher 148 4. Remove the filter function from ResponseStressc++Provider. 149
150 karl 1.1.2.16 21 November 2013 151 1. Mergeout from head of tree to 21 November 2013.
152 karl 1.1.2.15 153 18 November 2013 154 1. Cleanup of a bunch of minor errors and completion of all of the code for 155 the openQueryInstances except for the PullInstances in Dispatcher and 156 the aggregator function. 157 2. OpenqueryInstances added to cimcli.
158 karl 1.1.2.14 159 13 October 2013 CVS branch update. 160 1. Integrated bug 9786 into the branch. Note that we need to test the 161 generated statistics. 162 2. Mergeout executed to update to head of tree as of 8:00 am 13 October 2013. 163 3. Cleaned up several errors in OOP processing. Note that there is at least 164 one issue left when we to a pull on ManagedElement in at least one of the 165 namespaces. 166 4. Cleaned up some of the outstanding diagnostic code 167 5. Generally passes all tests except for one test of pullop where it is trying 168 to pull enum instances CIM_ManagedElement from a particular namespace.
169 karl 1.1.2.12
170 karl 1.1.2.16 NOTE: I did not make comments here for changes in October despite the fact 171 that I did 2 mergouts, number of fixes, and a mergein. 172
173 karl 1.1.2.12 30 September 2013 - CVS Update 174 Mergeout head of tree up to 29 September 2013.
175 karl 1.1.2.10 176 29 September 2013. CVS update. 177 1. Modified calls to statisticalData.cpp to a) directly call with request 178 type, b) incorporate the open, pull, etc. messages. However, since these 179 are not part of the CIM class, we must do something special with them.
180 karl 1.1.2.13 See bug 9785 for full solution to this issue.
181 karl 1.1.2.10 2. Corrected OOP interface to enable new flag to indicate internal operations 182 and set host, etc. 183 3. Add code to CQLOperationsDispatcher and WQLOperationDispatcher to clean 184 up CIMResponseDataCounter after filtering. 185 4. Modified ProviderAgent to set Host info for some pull operations. 186 5. Added new flag to CIMBinMsgSerializer and Deserializer.
187 karl 1.1.2.8
188 karl 1.1.2.9 17 September 2013 CVS update (Actually two different updates over 3 days) 189 1. Clean up some issues in CIMMessage.h and CIMMessage.cpp 190 2. Extend OpenExecQuery to WQL and CQL processors but return not complete 191 3. Remove memory leak in EnumerationContext and EnumerationContextTable 192 handling. 193 4. Created template functions for much of the pull operations. 194 5. Reversed order of queryLanguage and query (and changed names to match 195 execQuery) in client and server. Note that these are the execQuery 196 WQL and CQL filters and NOT FQL filters. 197 6. Some code cleanup in dispatcher 198 7. Today, passes all tests in pullop but issue in alltests. For some reason 199 not finding CIMObjectManager instance. Also, leaves enumeration contexts 200 if client terminates since cleanup thread not operating.
201 karl 1.1.2.11 8. XML from OOP not correctly processed.
202 karl 1.1.2.9
203 karl 1.1.2.8 14 September 2013 CVS update 204 Merged out up to 25 August. Cleaned up all operations and standardized code. 205 At this point the non pull operations code is in a set of templates but the 206 pull is not yet. 207 Fixed a significant number of problems so that it appears that the operations 208 except for OpenExecQuery run stably, at least with the pullop test program. 209 Note that there is a problem in that the Interop control provider is not 210 returning its singleton wbemserver object for some reason. Causes a test 211 failure
212 karl 1.1.2.7 213 Fixed for 16 June CVS Update 214 1. Cleaned up the enumerationContext and Table release functions and tested 215 to confirm that we do not lose memory in either normal sequences or 216 sequences that close early. Cleaned up pullop and added more tests
217 karl 1.1.2.8 Taged Before: PREAUG25UPDATE and after POSTAUG25UPDATE
218 karl 1.1.2.4 219 Fixed for 9 June CVS update 220 1. Cleaned up code for OpenQueryInstances. Note that this is incomplete. 221 No support in WQL or CQL Operations 222 2. 223 224 What was fixed for 5 June checkin.
225 karl 1.1.2.3 1. Extended ResponseTest MOF for for both CMPI and C++ subclasses 226 2. Fixed issues with pullop. 227 3. Fixed temp issue with CIMResponseData size by putting in mutex. That 228 is not a permanent fix but it gets around issue probably in the control 229 of the move logic that meant counts were off. 230 4. Fixed issues in Dispatcher so that associator code works. Still messy 231 code in the dispatcher. 232 5. Changed name of Enumerationtable.h & cpp to EnumerationContextTable.* 233 6 Changed name of ResponseStressTest module, classes, etc. 234 235 TAG: TASK_PEP317_5JUNE_2013_2 236
237 karl 1.1.2.2 2 June 2013
238 karl 1.1.2.1 239 Issues - KS 240
241 karl 1.1.2.15 - Still way to many TODO and KS comments and KS_TEMPS. Removing bit by bit.
242 karl 1.1.2.1
243 karl 1.1.2.15 - Runtime variable connection for the config parameters not installed. That 244 has been made into a separate bug (see bug 9819)
245 karl 1.1.2.1 246 5. Issue with the threaded timer. For some reason during tests it 247 eventually calls the timer thread with trash for the parm (which is 248 pointer to the EnumerationTable object). Caught because we do a valid 249 test at beginning of the function. 250
251 karl 1.1.2.2 6. Still using the templates in CIMOperationRequestDispatcher to simplify 252 the handle... processing. 253 254 7. I think I have a way around the double move of objects in the 255 EnumerationContext so that the outputter will just take a defined number 256 of objects directly from the gathering cache and save the second move. 257
258 karl 1.1.2.15 8. Not yet passing all tests but getting closer now. The major test that is 259 causing an error today is the execution of a full enumeration with the 260 forceProviders = true. This causes a client timeout sometimes.
261 karl 1.1.2.2 262 263
264 karl 1.1.2.1 =========================================== 265 266 OVERVIEW: 267 268 The operation extensions for pull operations defined in the DMTF specification 269 DSP0200 V 1.4 were implemented in Pegasus effective Pegasus version 2.11 270 including Client and Server. 271 272 These operations extend the CIM/XML individual operations to operation 273 sequences where the server must maintain state between operations in a 274 sequence and the client must execute multiple operations to get the full 275 set of instances or instance paths. 276 277 The following new CIM/XML operations as defined in DSP0200 are included; 278 279 -OpenEnumerateInstances 280 -openEnumerateInstancePaths 281 -OpenReferenceInstances 282 -OpenReferenceInstancePaths 283 -OpenAssociatiorInstances 284 -OpenAssociatorInstancePaths
285 karl 1.1.2.16 -OpenQueryInstances
286 karl 1.1.2.1 -PullInstancesWithPath 287 -PullInstancePaths
288 karl 1.1.2.16 -PullInstances
289 karl 1.1.2.1 -CloseEnumeration 290 -EnumerationCount
291 karl 1.1.2.2 OpenExecQuery
292 karl 1.1.2.1 293 The following operations have not been implemented in this version of Pegasus: 294 295 -OpenQueryInstances 296 297 The following limitations on the implementation exist; 298 299 1. The filterQueryLanguage and filterQuery parameters are processed by 300 the Pegasus client but the server returns error if there is any data in
301 karl 1.1.2.2 either parameter. This work does not include the development of the 302 query language. Note that a separate effort to extend Pegasus to use 303 the DMTF FQL query language is in process.
304 karl 1.1.2.1 305 2. The input parameter continueOnError is processed correctly by the client 306 but the Pegasus server only provides for false since the server does not 307 include logic to continue processing responses after an error is 308 encountered. 309 This is consistent with the statement in the specification that use of 310 this functionality is optional and the fact that the DMTF agrees that all 311 of the issues of continuing after errors have not been clarified. 312 313 3. The operation enumerationCount is not processed by the server today since 314 a) really getting the count would be the same cost as the corresponding 315 enumeration, b) the server does not include a history or estimating 316 mechanism for this to date.
317 karl 1.1.2.2 NOTE: After a through review as part of the development of the next version 318 of CMPI we have concluded that this operation is probably not worth the 319 effort. Since it is optional, Pegasus will only return the unknown status 320 at this point
321 karl 1.1.2.1 322 Since the concept of sequences of operations linked together (open, pull, close) 323 is a major extension to the original CIM/XML operation concept of completely 324 independent operations several new pieces of functionality are implemented 325 to control interOperationTimeouts, counts of objects to be returned, etc. 326 327 TBD - Review this 328 329 CLIENT 330 331 The new operations follow the same pattern as the APIs for existing operations 332 in that: 333 334 1. All errors are handled as CIMException and Exception 335 336 2. The means of inputting parameters are the same except that there are 337 significantly more input parameters with the open operations and for the 338 first time operations return parameters as well as objects in the 339 response. Specifically the open and pull operations return values for 340 enumerationContext which is the identity for a pull sequence and 341 endOfSequence which is the marker the server sends in open and pull 342 karl 1.1.2.1 responses when it has no more objects to send. 343 344 The significant differences include: 345 346 1. Processing of parameters on responses (i.e. the endOfSequence and 347 enumerationContext parameters are returned for open and pull operations). 348 349 2. Numeric arguments (Uint32 and Uint64 include the option of NULL in some 350 cases so they are packaged inside classes Uint32Arg and Uint64Arg in the 351 client api. 352 353 3. The association and reference operations ONLY process instances. They do 354 not include the capability to return classes like reference and associator 355 do and therefore return CIMInstance rather than CIMObject. 356 357 4. Paths are returned in all cases (i.e OpenEnumerateInstances and 358 PullInstancesWithPath where they were not with EnumeratInstances. 359 360 5. The client must maintain state between operations in a sequence (using 361 the enumerationContext parameter). 362 363 karl 1.1.2.1 TBD- Are there more differences. 364 365 366 SERVER 367 368 The Pegasus server attempts to always deliver the requested number of objects 369 for any open or pull request (the specification allows for the server to 370 deliver less than the requested number of objects and specifically to return 371 zero objects on open). We felt that it was worth any extra cost in processing 372 to provide the client with exactly what it had requested. 373 374 The pegasus server always closes an enumeration sequence upon receipt of any 375 error from the providers, repository, etc. Therefore the server will reject
376 karl 1.1.2.2 any request that has continueOnError = true; 377 378 Expansion to allow the continue on error may be added in a future version. 379 In any case, the whole purpose of the continue on error is really to allow 380 input from good providers to be mixed with providers that return errors so 381 that generally this would mean simply changing the logic in the return mechanism 382 to not shutdown when an error is received from any given provider. 383 384 Generally we do not believe that the providers need to do much more in the 385 future to support the continueOnError other than possibly allowing the provider 386 to continue processing after it has received an error.
387 karl 1.1.2.1 388 PROVIDERS 389 390 This implementation requires NO changes to the existing providers. The 391 provider APIs operate just as they do with the original operations. 392 393 Because the server processing is different however, there may be some 394 behavior differences primarily because the client now controls the speed of 395 delivery of objects. 396 397 In previous versions of Pegasus, the server attempts to deliver objects as 398 rapidly as then can be put on the network. In the case of HTTP chunked requests 399 they are delivered in chunks of about 100 objects. The primary delay for the 400 providers was the processing of each segment through the server. The server 401 is blocked so that no other segment can proceed through the server until that 402 segment is processed and sent on the network. 403 In the case of non-chunkedresponses, they are completely gathered in the serve 404 and then delivered as one non-chunked response. There were no delays for the 405 providers, just lots of possible memory use in the server. 406 407 The responses from providers (delivered through the deliver(...) interface are 408 karl 1.1.2.1 gathered into segments of about 100 objects and this group of objects is moved 409 through the server to be delivered to the client. 410 411 However with the inclusion of the pull operations, The segments of objects 412 from the providers are cached in the server response path until the 413 maxObjectCount for that request (open or pull) and that number returned in a 414 non-chunked response. Thus, if the client is slow to issue pull requests, 415 the providers might be delayed at some point to reduce memory usage in the 416 server (the delay appears as slow response tothe deliver operation). 417 418 In other words, the time to process large sets of responses from the provider 419 now depends on the speed of handling the client. 420 421 It is important to remember in developing providers that the Pegasus server 422 can most efficiently process responses if they are passed from the provider 423 to the server individually or in small arrays of objects rather than the 424 provider gathering very large arrays of objects and sending them to the 425 server. 426
427 karl 1.1.2.2 NEXT GENERATION PROVIDERS 428 KS_TODO 429
430 karl 1.1.2.1 CONFIGURATION PARAMETERS 431 432 The server includes several configuration parameters to set limits on the 433 processing of pull operations. All of these configuration parameters are 434 compile time parameters rather than runtime. 435 436 1. Maximum value of minimum interoperation time. This parameter defines the 437 maximum time allowed between the return of an open or pull response and 438 the receipt of the next pull or a close operation before the server may 439 close the enumeration. The specification allows the server to set a 440 maximum interoperation time and refuse open requests that with requested 441 operationTimeout greater than that time. 442 CIM_ERR_INVALID_OPERATION_TIMEOUT 443 444 This value is set with the Pegasus environment variable 445 PEGASUS_PULL.... 446 447 2. Maximum objects returned in a single open or pull operation. The server 448 can set a maximum limit on the number of objects that can be returned in 449 a single open or pull oepration with the maxObjectCount parameter. 450 451 karl 1.1.2.1 3. Whether the server allows 0 as an interoperation timeout value. The value 452 zero is s special value for the interoperationTimeout in that it tells the 453 server to not timeout any enumeration sequence. 454 455 With this value for interoperationTimeout, the only way to close an 456 enumeration sequence is to complete all of the pulls or issue the close. 457 If for some reason the sequence is not completed, that enumeration context 458 would remain open indefinitly. Since in Pegasus any open enumeration 459 context uses resources (the context object and any provider resposnes that 460 have not yet been issued in a response) it would appear that most 461 platforms would not want to allow the existence of enumeration contexts 462 that cannot be closed by the server. 463 464 4, maximum consecutive pull requests with 0 maxObjectCount. The use of the 465 pull operation with maxObjectCount set to zero could be used to keep an 466 enumeration context open indefinitly (this tells the server to restart the 467 interoperationTimeout but not send any objects in the response). Therefore the 468 specification allows for the server setting maximum limits on this behavior 469 and returning the error CIM_ERR_SERVER_LIMITS_EXCEEDED if this limit is 470 exceeded. 471 Note that this is maximum CONSECUTIVE pulls so that issuing a pull with 472 karl 1.1.2.1 a non-zero count resets this counter. 473 474 KS-TBD - Is this really logical since we can still block by just issuing 475 lots of zero request and an occansional request for one object. 476 477 Pegaus sets the value of this limit to 1000 and allows the implementer to 478 modify it with the PEGASUS_MAXIMUM_ZERO_OBJECTCOUNT environment variable. 479 480 5. Default operationTimeout - 481 482 The default of this parameter is to refuse operat 483 484 In the current release of Pegasus these are all compile time parameters.
485 karl 1.1.2.11 486 487 NOTES On working with task branch. 488 489 Merge out Process 490 491 To keep our TASK branch in sync with the current head of tree we need 492 to do a regular merge out. the TaskMakefile contains the makefile 493 procedures to do this efficiently. NOTE: Following these procedures is 494 important in that you are merging out new material each time you do 495 the merge out. If you were just to repeatedly merge out, you would be 496 merging previously merged changes a second time causing a real mess. 497 498 Start with new directory and put TaskMakefile above pegasus (needed so you 499 have this file for the initial operations. 500 501 make -f TaskMakefile branch_merge_out BNAME=PEP317-pullop ## takes a long time 502 503 This checks out current head, merges it into task branch and sets tags 504 for the mergeout. Note that at the end of this step this work is 505 part of the TASK... branch. 506 karl 1.1.2.11 507 NOW check for conflicts, errors, etc. that resulted from the merge. 508 Look for conflict flags, compare the results (I use linux merge as a 509 good graphic compare tool) and build and test. When you are satisfied 510 that the merge out is clean, you can commit the results to the TASK... 511 branch 512 513 To commit the work to this into Task branch 514 515 make -f mak/TaskMakefile branch_merge_out_commit BNAME=PEP317-pullop 516 517 or manually commit and finish as follows 518 519 cvs commit 520 make -f mak/TaskMakefile branch_merge_out_finish BNAME=PEP317-pullop 521 522 ## This last step is important since it cleans up temporary tags to prepare 523 you for the next checkout 524 525 COMPARE TASKBRANCH WITH HEAD 526 527 karl 1.1.2.11 In a new pegasus work space do same as above for merge out. 528 529 make -f TaskMakefile BNAME=PEP317-pullop 530 531 This produces a result which is all of the head merged into the branch. 532 A diff of this is all the new changes to the head of tree that you will 533 include into the merge. 534

No CVS admin address has been configured