version 1.1.2.8, 2013/09/14 23:08:12
|
version 1.1.2.26, 2014/04/23 00:49:58
|
|
|
checkin to head of tree. Please feel free to add notes, etc in this | checkin to head of tree. Please feel free to add notes, etc in this |
section as you review/test.>>>>>> | section as you review/test.>>>>>> |
| |
NOTES On working with task branch. |
|
|
|
Merge out Process |
|
|
|
To keep our TASK branch in sync with the current head of tree we need |
|
to do a regular merge out. the TaskMakefile contains the makefile |
|
procedures to do this efficiently. NOTE: Following these procedures is |
|
important in that you are merging out new material each time you do |
|
the merge out. If you were just to repeatedly merge out, you would be |
|
merging previously merged changes a second time causing a real mess. |
|
|
|
Start with new directory and put TaskMakefile above pegasus (needed so you |
|
have this file for the initial operations. |
|
|
|
make -f TaskMakefile branch_merge_out BNAME=PEP317-pullop ## takes a long time |
|
|
|
This checks out current head, merges it into task branch and sets tags |
|
for the mergeout. Note that at the end of this step this work is |
|
part of the TASK... branch. |
|
|
|
NOW check for conflicts, errors, etc. that resulted from the merge. |
|
Look for conflict flags, compare the results (I use linux merge as a |
|
good graphic compare tool) and build and test. When you are satisfied |
|
that the merge out is clean, you can commit the results to the TASK... |
|
branch |
|
|
|
To commit the work to this into Task branch |
|
|
|
make -f mak/TaskMakefile branch_merge_out_commit BNAME=PEP317-pullop |
|
|
|
or manually commit and finish as follows |
|
|
|
cvs commit |
|
make -f mak/TaskMakefile branch_merge_out_finish BNAME=PEP317-pullop |
|
|
|
## This last step is important since it cleans up temporary tags to prepare |
|
you for the next checkout |
|
|
|
COMPARE TASKBRANCH WITH HEAD |
|
|
|
In a new pegasus work space do same as above for merge out. |
|
|
|
make -f TaskMakefile BNAME=PEP317-pullop |
|
|
|
This produces a result which is all of the head merged into the branch. |
|
A diff of this is all the new changes to the head of tree that you will |
|
include into the merge. |
|
|
|
|
|
TODO list: | TODO list: |
1. Binary operation from OOP. Need to add counter to binary | 1. Binary operation from OOP. Need to add counter to binary |
protocol to be able to count objects in response. Generates | protocol to be able to count objects in response. Generates |
warnings in things like messageserializer and does not work with | warnings in things like messageserializer and does not work with |
OOP right now. |
OOP right now. Corrected by converting to XML. |
2. OpenExecQuery - Code is incorrect in that it used InstancesWithPath |
2. OpenExecQuery - Code is incomplete in that it does not include the |
where the spec is instances with no path. Need new function to wrap |
return from the exec query function to the aggregator yet. |
getInstanceElement(withoutPathElement) in XmlReader. Note that |
|
Alternate is to put flag on InstancesWith Path to say no path |
|
3. Code for Pull part of OpenQueryInstancesRequest a) should be part of | 3. Code for Pull part of OpenQueryInstancesRequest a) should be part of |
the common CIMOperationRequestDispatcher execCommon code. | the common CIMOperationRequestDispatcher execCommon code. |
4. The changes to WQLCIMOperationRequestDispatcher and CQL... for handling | 4. The changes to WQLCIMOperationRequestDispatcher and CQL... for handling |
pull not completed so we feed the responses back to the EnmerationContext | pull not completed so we feed the responses back to the EnmerationContext |
queues | queues |
3. Lots of minor TODOs, diagnostics, etc. |
3. Lots of minor TODOs, diagnostics, etc. still in the code |
4. External runtime variables. Decide this as part of PEP. The variables |
4. External runtime variables. Proposing that they be fixed for this release |
exist in CIMOperationRequestDispatcher but not in CIMConfig. The primary |
rather than set by configuration. This should be discussed. Am making |
ones to consider are: |
this a separate bug. See bug 9819 for the changes to cover this. |
a. System maxObjectCount. Setting some maximum size on what a pull |
5. Consider moving some more of the code in dispatcher from templates to |
client can request (i.e. the maximum size of the maxObjectCount on |
common functions which would mean adding intermediate classes in |
Open... and pull operations. |
CIMMessage but would reduce code size. Currently the change that |
b. Pull interoperationTimeout (max times between operations). This is |
we should make for this is to create a new message CIMPullRequestData... |
the maximum number of seconds on the operationTimeout parameter of the |
between CIMRequestDataMessage and the Pull requests since they all |
Open operations |
have exactly the same input and output pararameters (The |
c. Maximum size of the responseCache before it starts backing up |
CIMResponseData hides the response object differences) |
responses to the providers. |
6. Extension to avoid double move of objects in CIMResponseData (one |
5. Decision on EnumerationContext timeout (separate thread or just |
|
checks during other operations). Can we, in fact really keep the |
|
enumeration context table and queue under control without monitoring |
|
with a separate thread. We must monitor for: |
|
a. Client operation that stop requesting (i.e. inter operation time |
|
exceeds operationTimeout). Note that if it simply exceeds the time |
|
the next operation does the cleanup. The issue is those clients that |
|
simply stop and do not either close or go to completion. |
|
b. We should protect against providers that no not every finish delivering |
|
or take to long between deliveries. This does not exist in Pegasus |
|
today |
|
6. Clean up code in Dispatcher. Want to at least reduce the code for the |
|
Open Operations to a set of templates so we know that the code is the |
|
same for all operations. Right now the existing operations enum, assoc, |
|
etc. are all in templates but not the open operations. Note that the |
|
pull is a single template for both pullInstances and pullInstancePaths. |
|
However, we might be able to reduce this to a single function by adding |
|
a new level to CIMMessage.h (CommonPullResponse) |
|
7. Extension to avoid double move of objects in CIMResponseData (one |
|
into enumerationContext queue and second to new cimResponseData for | into enumerationContext queue and second to new cimResponseData for |
response. Want to avoid second move by extending Open/Pull response | response. Want to avoid second move by extending Open/Pull response |
messages to include count and CIMResponse data to count objects out | messages to include count and CIMResponse data to count objects out |
of queue when converting (avoids the second move). Big issue here |
of queue when converting (avoids the second move). This would mean |
with binary data since need to extend format to count it. |
extending the output writers to allow the count field to be supplied |
8. NEXT TASKS: |
so they would only create output for up to the count supplied. |
a. get the pull operations rather than a template into a single |
9. Add more static tests (currently only OpenEnumerateInstances is covered). |
function by creating a new CIMPullResponse message in CIMMessage.h that |
Low priority because most of this except exact XML covered in other |
contains the pull data. Then we can use a single function to process all |
tests. |
pull operations. |
10. Could clean up the CIMOperationRequestDecoder.cpp rejectCIMException |
b. test the timeout thread |
further since there is some overlap. That is low priority. |
c. New Mergout to bring up to to date again. |
12.Add exception for to many entries in EnumerationContextTable. The |
d. Fix known problem with the interop provider. |
test exists but is ignored right now. |
|
12. Correct issue between operations and HTTP where we are sending |
|
trailers with exceptions. Also correct issue where we are |
|
sending endOfSequence and EnumerationContext in response when |
|
we send CIMException. |
|
|
|
NEXT TASKS: |
|
b. Finish and test the OpenQueryInstances |
|
c. Clean up TODOs |
|
d. Find issue that is causing occasional failures under load. It |
|
is probably a threading issue between the 3 possible threads |
|
that use the EnumerationContext (request thread, provider thread, |
|
timeout thread) and a place where not mutuxed, primarily the |
|
timeout thread. |
|
|
|
22 April 2014 |
|
1. Clean up some messages. |
|
2. Still trying to find issue that randomly keeps messages in timeout |
|
queue long after they have been closed and returned. |
|
3. Fixed memory loss error. |
|
4. Added zero reject for OperationTimeout because that effectively disables |
|
the whole timeout mechanism so that enumerations could be started, never |
|
completed by the client and would just hang around in memory. That |
|
would be a significant DDOS attack (start very large enum with no |
|
timeout) and never do pulls after open. |
|
|
|
21 April 2014 |
|
1. Reorganizzed code in CIMOperationRequestDecoder.cpp ...Iparam classes. |
|
Cleaned up the rejectCIMException code. |
|
2. Cleanup some issues in the EnumerationContextTimeout Thread. |
|
3. Fixed issue in pullop where it was using zero as default for default |
|
operation timeout where it should have been NULL, the DMTF specified |
|
default. |
|
4. Enabled the test to reject 0 as operation timeout on a request and but |
|
controled by a #define in CIMOperationRequestDispatcher.cpp |
|
5. Added test for excessive new OperationContext entries in table. |
|
|
|
18 April 2014 |
|
1. Internationalize remaining messages in CIMOperationRequestDecoder.cpp and |
|
clean up the decoders for some of the messages so they use the |
|
...Iparam classes. Note that we did not clean up all of the old message |
|
decoders, just the new ones and we did not clean up the Count message |
|
since it is deprecated. TODO clean these up so the decoder is really |
|
consistent. |
|
2. Enabled the EnumerationContext Timer and removed code for alternate |
|
non-thread solution. |
|
|
|
17 April 2014 |
|
1. Reorganized the lock/unlock for handling the EnumerationContext because |
|
there were thread windows (i.e. using the context after it was released |
|
by other thread). |
|
2. Modified code for the InteroperationTimerThread and deleted old code that |
|
allowed this whole thread to by bypassed. |
|
3. Internationalized a number of CIMOperationRequestDispatcher CIMException |
|
response messages. |
|
4. Fixed stress tests in Client/tests/pullop that were getting timeout. The |
|
issue was not the pull code but that fact that the idle client timeout |
|
appears to be measureing to the end of the operation and the operation |
|
was taking more than 40 seconds which is the default timeout. The idle |
|
timeout takes no account of the fact that chunks are being regularly |
|
delivered. |
|
5. Several minor code changes. |
|
NOTE: At this point the code passes extensive tests of the pull operations |
|
both in OOP and none OOP mode (but with lots of diagnostics that may |
|
be affecting timing). |
|
|
|
10 April 2014 |
|
1. Fix issues causing failure with repeated pull operations. |
|
2. Consolidate some of the CIMOperationRequestDispatcher.cpp template code |
|
|
|
7 April 2104 |
|
MergeOut of head of tree. |
|
|
|
7 April 2014 |
|
1. Permanently fixed issue with getting correct number of objects on response. |
|
This was introduced when did a temp fix to the OOP processing, in particular |
|
removed the condition variable wait in the CIMOperationDispatcher handle |
|
functions for opens and pulls. This moved the functionality to kick off |
|
open and pull responses to the provider threads if there are not enough |
|
objects when the handle is processed. Also added a variable to allow us to |
|
test with either a) responses are required to satisfy the original |
|
request size or) b, responses are required only to return some objects. |
|
We will add statistics to see which of these works best. |
|
2. Modified WsmProcessor EnumerationContext class since the name conflicts |
|
with the pull Operation EnumerationContext class. |
|
3. Created a new intermediate level of CIMMessage, the CIMPullResponseData |
|
Message so that all of the open and Pull responses can use common code |
|
since there are only two variables (endOfSequence and EnumerationContext) |
|
and they are common across all the open and pull responses. |
|
4. Removed a number of diagnostics. |
|
5. Cleaned up the Dispcatcher so that the open and pulls have common response |
|
code and that code can be used from the dispatcher handle functions and |
|
the provider response functions. |
|
|
|
31 March 2014 - Checkin |
|
1. Fixed issues in OOP processing of pull operations, in particular |
|
issues with cimxml output format when processed through the |
|
*InternalXmlEncoder functions. |
|
2. Clean up some of the internalXml functionality |
|
3. Found issues causing timeout with a particular provider. The issue |
|
is that the dispatcher and monitor end up using the same thread so the |
|
condition variable in the dispatcher thread stops the monitor. Turned |
|
off the conditionVariable in getCache for the moment which means that |
|
we get number of responses for open... with 0 objects before the |
|
providers can begin to respond. This is only for test. |
|
4. Added some statistics for enumerations and display the statistics |
|
when we close the server (same as cache statistics) |
|
|
|
12 March 2014 - Mergeout and Mergein |
|
1. Mergeout to head of tree for this date and mergein for patch update |
|
to bug 9676 |
|
2. Extensions to pullop tests program and tests. |
|
3. Added some diagnostics in looking for OOP issue. |
|
4. Removed a number of diagnostics messages and cleaned up code in |
|
dispatcher to simplify pull operation processing. |
|
|
|
15 December 2013 |
|
1. Mergeout and mergein up to 15 December 2013 |
|
2. Clean up issues from tests documented in bug 9676 last week. |
|
3. Clean up some code in dispatcher |
|
4. Remove the filter function from ResponseStressc++Provider. |
|
|
|
21 November 2013 |
|
1. Mergeout from head of tree to 21 November 2013. |
|
|
|
18 November 2013 |
|
1. Cleanup of a bunch of minor errors and completion of all of the code for |
|
the openQueryInstances except for the PullInstances in Dispatcher and |
|
the aggregator function. |
|
2. OpenqueryInstances added to cimcli. |
|
|
|
13 October 2013 CVS branch update. |
|
1. Integrated bug 9786 into the branch. Note that we need to test the |
|
generated statistics. |
|
2. Mergeout executed to update to head of tree as of 8:00 am 13 October 2013. |
|
3. Cleaned up several errors in OOP processing. Note that there is at least |
|
one issue left when we to a pull on ManagedElement in at least one of the |
|
namespaces. |
|
4. Cleaned up some of the outstanding diagnostic code |
|
5. Generally passes all tests except for one test of pullop where it is trying |
|
to pull enum instances CIM_ManagedElement from a particular namespace. |
|
|
|
NOTE: I did not make comments here for changes in October despite the fact |
|
that I did 2 mergouts, number of fixes, and a mergein. |
|
|
|
30 September 2013 - CVS Update |
|
Mergeout head of tree up to 29 September 2013. |
|
|
|
29 September 2013. CVS update. |
|
1. Modified calls to statisticalData.cpp to a) directly call with request |
|
type, b) incorporate the open, pull, etc. messages. However, since these |
|
are not part of the CIM class, we must do something special with them. |
|
See bug 9785 for full solution to this issue. |
|
2. Corrected OOP interface to enable new flag to indicate internal operations |
|
and set host, etc. |
|
3. Add code to CQLOperationsDispatcher and WQLOperationDispatcher to clean |
|
up CIMResponseDataCounter after filtering. |
|
4. Modified ProviderAgent to set Host info for some pull operations. |
|
5. Added new flag to CIMBinMsgSerializer and Deserializer. |
|
|
|
17 September 2013 CVS update (Actually two different updates over 3 days) |
|
1. Clean up some issues in CIMMessage.h and CIMMessage.cpp |
|
2. Extend OpenExecQuery to WQL and CQL processors but return not complete |
|
3. Remove memory leak in EnumerationContext and EnumerationContextTable |
|
handling. |
|
4. Created template functions for much of the pull operations. |
|
5. Reversed order of queryLanguage and query (and changed names to match |
|
execQuery) in client and server. Note that these are the execQuery |
|
WQL and CQL filters and NOT FQL filters. |
|
6. Some code cleanup in dispatcher |
|
7. Today, passes all tests in pullop but issue in alltests. For some reason |
|
not finding CIMObjectManager instance. Also, leaves enumeration contexts |
|
if client terminates since cleanup thread not operating. |
|
8. XML from OOP not correctly processed. |
| |
14 September 2013 CVS update | 14 September 2013 CVS update |
Merged out up to 25 August. Cleaned up all operations and standardized code. | Merged out up to 25 August. Cleaned up all operations and standardized code. |
|
|
2 June 2013 | 2 June 2013 |
| |
Issues - KS | Issues - KS |
1. have not installed the binary move in CIMResponseData. Please run |
|
with OPP off. |
|
2. Some problem in the processing so we are getting server crashes. |
|
Right no I am guessing that this is in the binaryCodec and am going to |
|
expand the test tools to allow testing through the localhost. |
|
| |
3. Still way to many TODO and KS comments and KS_TEMPS. Removing bit by bit. |
- Still way to many TODO and KS comments and KS_TEMPS. Removing bit by bit. |
| |
4. Env variable connection for the config parameters not installed. |
- Runtime variable connection for the config parameters not installed. That |
|
has been made into a separate bug (see bug 9819) |
| |
5. Issue with the threaded timer. For some reason during tests it | 5. Issue with the threaded timer. For some reason during tests it |
eventually calls the timer thread with trash for the parm (which is | eventually calls the timer thread with trash for the parm (which is |
|
|
EnumerationContext so that the outputter will just take a defined number | EnumerationContext so that the outputter will just take a defined number |
of objects directly from the gathering cache and save the second move. | of objects directly from the gathering cache and save the second move. |
| |
8. Not yet passing all tests but getting closer now. |
8. Not yet passing all tests but getting closer now. The major test that is |
|
causing an error today is the execution of a full enumeration with the |
9. Created a tag before this commit TASK_PEP317_1JUNE_2013. |
forceProviders = true. This causes a client timeout sometimes. |
| |
10. Next Tag will be TASK_PEP317_2_JUNE_2013 in the task branch |
|
| |
| |
=========================================== | =========================================== |
|
|
-OpenReferenceInstancePaths | -OpenReferenceInstancePaths |
-OpenAssociatiorInstances | -OpenAssociatiorInstances |
-OpenAssociatorInstancePaths | -OpenAssociatorInstancePaths |
|
-OpenQueryInstances |
-PullInstancesWithPath | -PullInstancesWithPath |
-PullInstancePaths | -PullInstancePaths |
|
-PullInstances |
-CloseEnumeration | -CloseEnumeration |
-EnumerationCount | -EnumerationCount |
OpenExecQuery | OpenExecQuery |
|
|
The default of this parameter is to refuse operat | The default of this parameter is to refuse operat |
| |
In the current release of Pegasus these are all compile time parameters. | In the current release of Pegasus these are all compile time parameters. |
|
|
|
|
|
NOTES On working with task branch. |
|
|
|
Merge out Process |
|
|
|
To keep our TASK branch in sync with the current head of tree we need |
|
to do a regular merge out. the TaskMakefile contains the makefile |
|
procedures to do this efficiently. NOTE: Following these procedures is |
|
important in that you are merging out new material each time you do |
|
the merge out. If you were just to repeatedly merge out, you would be |
|
merging previously merged changes a second time causing a real mess. |
|
|
|
Start with new directory and put TaskMakefile above pegasus (needed so you |
|
have this file for the initial operations. |
|
|
|
make -f TaskMakefile branch_merge_out BNAME=PEP317-pullop ## takes a long time |
|
|
|
This checks out current head, merges it into task branch and sets tags |
|
for the mergeout. Note that at the end of this step this work is |
|
part of the TASK... branch. |
|
|
|
NOW check for conflicts, errors, etc. that resulted from the merge. |
|
Look for conflict flags, compare the results (I use linux merge as a |
|
good graphic compare tool) and build and test. When you are satisfied |
|
that the merge out is clean, you can commit the results to the TASK... |
|
branch |
|
|
|
To commit the work to this into Task branch |
|
|
|
make -f mak/TaskMakefile branch_merge_out_commit BNAME=PEP317-pullop |
|
|
|
or manually commit and finish as follows |
|
|
|
cvs commit |
|
make -f mak/TaskMakefile branch_merge_out_finish BNAME=PEP317-pullop |
|
|
|
## This last step is important since it cleans up temporary tags to prepare |
|
you for the next checkout |
|
|
|
COMPARE TASKBRANCH WITH HEAD |
|
|
|
In a new pegasus work space do same as above for merge out. |
|
|
|
make -f TaskMakefile BNAME=PEP317-pullop |
|
|
|
This produces a result which is all of the head merged into the branch. |
|
A diff of this is all the new changes to the head of tree that you will |
|
include into the merge. |
|
|