1 j.alex 1.2 ===============================================================================
2 Please See PEP# 241 for more details on Open Pegasus Stress Tests:
3 ===============================================================================
4
|
5 kumpf 1.3 Open Pegasus stress tests are expected to exert excessive loads on the CIM
6 server. The main intent of these tests are to see how fast, or whether, we can
7 get the CIM server to crash or hang. These tests are designed to allow some
|
8 j.alex 1.2 flexibility in adjusting the desired stress level on the CIM server.
9
10 ===============================================================================
11 Table of Contents:
12 ===============================================================================
13 1.0 Execution model assumptions
14 2.0 OpenPegasus Stress Test Overview
15 2.1 TestStressTestController
16 2.2 Stress Test Clients
17 2.3 The Stress Test Configuration File
18 3.0 Default Values
19 4.0 Running a Stress Test using default values
20 5.0 Running the new "stresstest" target
21 6.0 Stress Test Logs
22 7.0 Stress Test Errors
23 8.0 Other Considerations
24 ===============================================================================
25
26 *******************************************************************************
27 1.0 Execution model assumptions:
28 ===============================================================================
29 j.alex 1.2 1.1 The proposed client-side functionality concerning stress test will be
|
30 kumpf 1.3 tested only on Windows and Linux. Note, that the CIM server that is
31 being stressed can be running on any of the OpenPegasus supported
|
32 j.alex 1.2 operating system.
33
34 1.2 The CIM server that is being stress tested is expected but not required
|
35 kumpf 1.3 to be running on a different system than the stress clients. This is
36 mainly to isolate the CIM server that is under stress.
37
|
38 j.alex 1.2 1.3 ALL OF THE PROPOSED STRESS TEST CLIENTS INVOKED BY A GIVEN EXECUTION OF
|
39 kumpf 1.3 THE "TestStressTestController" ARE EXPECTED TO BE STRESS TESTING THE SAME
|
40 j.alex 1.2 CIMSERVER.
41
42 ===================================================================i===========
|
43 kumpf 1.3 DIAGNOSIS OF CIM SERVER HANGS OR CRASHES CAUSED BY THE STRESS TESTS ARE
|
44 j.alex 1.2 EXPECTED TO BE DONE USING ALREADY EXISTING DEBUGGING AND TRACING TECHNIQUES
45 OR TOOLS.
46 ===============================================================================
47
48 *******************************************************************************
49 2.0 OpenPegasus Stress Test Overview:
50 ===============================================================================
|
51 kumpf 1.3 OpenPegasus Stress Test will be comprised of three major components, as
|
52 j.alex 1.2 follows:
53
54
|
55 kumpf 1.3 1. TestStressTestController
|
56 j.alex 1.2 Command that invokes one or more of the Stress Test Clients.
57
58 2. Stress Test Clients.
59 Client Programs that may contain one or more of the CIM Operation(s).
|
60 kumpf 1.3
|
61 j.alex 1.2 3. Stress Test Configuration.
62 A flat file containing test configuration information.
63
|
64 kumpf 1.3 The "stressTestController" will read the configuration file and then,
65 invoke the "Stress Test Client" instances specified in the "Stress Test
|
66 j.alex 1.2 Configuration". Each of the client instances will be a separate process.
|
67 kumpf 1.3 These Clients will in turn connect to the same CIM Server and perform
|
68 j.alex 1.2 their respective operations concurrently.
69
70 2.1 TestStressTestController:
71 =====================
72 This command can be run using the following options:
73 (Usage information may be displayed using "--help" option)
74
75 TestStressTestController [ -s ] [ -h hostname ] [ -p portnumber ]
76 [ -u username ] [ -w password ] [ --help ]
77 [ --version ] [ --verbose ] [<config_filename>]
78
79 Options :
80 -h - Connect to CIM Server on specified hostname
81 --help - Display this help message
82 -p - Connect to CIM Server on specified portnumber
83 -s - Use SSL protocol between 'stressTestController' client
84 and the CIM Server
85 -u - Connect to CIM Server using the specified username
86 --version - Display CIM Server version number
87 --verbose - Display verbose information
88 -w - Connect to CIM Server using the specified password
89 j.alex 1.2
90 Operands :
91 <config_filename>
92 - Specifies the name of the configuration file that is
93 to be used for the tests.
94
95 2.1.1 The user may specify the name of the configuration file that is to be
96 used for the tests. When a configuration file is not specified at
97 command line, then the TeststressTestController command will use the
98 default configuration file if it exists, otherwise take the default
99 values hard coded in the TestStressTestcontroller.
100
101 2.1.2 This command will also verify the validity of the configuration values
102 specified in the configuration file.
103
104 2.1.3 This command will invoke one or more of the Stress Test Clients
105 specified below.
106
107 2.1.4 This command will end for any of the following reasons:
108 - When "Duration" as specified in the Stress Test configuration file
109 has elapsed.
110 j.alex 1.2 - If the specified configuration file cannot be found in the
111 configuration directory.
112 - If the configuration file has any invalid parameter or value.
113 - A persistent failure in one (or more) of the Stress Test Clients.
114 - The configuration can be setup to allow some margin of client
115 failures. By default, any client failure will be reported
116 and result in termination of the stress test
117 (this corresponds to a ToleranceLevel of 0%).
118
|
119 kumpf 1.3 2.1.5 When the command ends, it will as part of its cleanup also end
|
120 j.alex 1.2 all the stress clients that it had originally started.
121
|
122 kumpf 1.3 2.1.6 All the options specified here on the command line will have
123 precedence over similar parameters specified in the configuration
|
124 j.alex 1.2 file.
125
126 2.1.7 Log files for the controller can be found in
127 $PEGASUS_HOME/test/StressTestController/log/ and will be named as
128 "<controller_pid>_stressTest_<DDMMYYhhmmss>.log "
129 where,
130 "DD" denotes the day.
131 "MM" denotes the month.
132 "YY" denotes the year.
133 "hh" denotes the hour.
134 "mm" denotes the minutes.
135 "ss" denotes the seconds.
136 "controller-pid" denotes the pid of the stressTestController
137
138 2.1.8 Log file for the clients intiated by a controller can be found in
139 $PEGASUS_HOME/test/StressTestController/log/ and will be named as
140 <controller-pid>_StressTestClients.log
141 where,
142 "controller-pid" denotes the pid of the stressTestController
|
143 kumpf 1.3
|
144 j.alex 1.2 2.2 Stress Test Clients:
145 ====================
|
146 kumpf 1.3
|
147 j.alex 1.2 The following table lists the Stress Test Clients and the associated CIM
148 Operations that each of these clients will perform on the CIM Server.
149 These clients once started will run repeatedly until stopped by the
150 stressTestController command. The expected stress factor imposed on the
151 CIM server is directly proportional to the number of client instances
152 running concurrently against the same CIM Server.
153
154 STRESS TEST CLIENTS DESCRIPTION
155
156 TestModelWalkStressClient For all the detectable namespaces this
157 client will perform an enumeration of
158 all Classes, Instances, References,
159 Associators & Qualifiers.
160
161 TestWrapperStressClient This is a wrapper client that will allow
162 existing clients or programs to be
163 existing clients or programs to be
164 executed. The intended client program
|
165 kumpf 1.3 that is expected to be executed must
166 be specified in the configuration file
|
167 j.alex 1.2 when using this client. The client program
|
168 kumpf 1.3 must ensure that it connect to the
|
169 j.alex 1.2 correct cimserver.
170 For example, this client can be used to
|
171 kumpf 1.3 run pegasus CLI's to stress test the CIM
172 server. Also note that the "CLI" command
|
173 j.alex 1.2 uses -l option to set the hostname where the
|
174 kumpf 1.3 CIMServer is running.
175 Please refer to section 2.3.8(A Sample
176 Configuration File) for an example.
|
177 j.alex 1.2
178 Note:
|
179 kumpf 1.3 The above list of stress test Clients can grow as needed.
|
180 j.alex 1.2 Please see PEP# 241 for possible future "Stress Test Clients":
181
|
182 kumpf 1.3 2.2.1 Stress Test Clients will Connect to the CIM Server running on the
183 specified hostname. However, if the hostname is not specified then
|
184 j.alex 1.2 the client will attempt to connect with the CIM server on the same
185 system the client is invoked on.
186
|
187 kumpf 1.3 2.2.2 "Stress Test Clients" on completion or failure of its operation as
|
188 j.alex 1.2 specified above will continue to repeat the operations until stopped
189 by the "TestStressTestController".
190
191 2.3 The Stress Test Configuration File:
192 ===================================
193
194 2.3.1 The default configuration filename for the stress test will be named
195 as "default_stresstest.conf".
|
196 kumpf 1.3
|
197 j.alex 1.2 2.3.2 The user may specify the name of the configuration file that is to be
198 used for the tests.
199
|
200 kumpf 1.3 2.3.3 Default configuration file will be kept in
|
201 j.alex 1.2 $PEGASSUS_ROOT/test/StressTestController/ directory.
202
203 2.3.4 The Configuration file can contain the following common values that
204 would in turn provide the factors required for the stress tests:
205
206 1. Duration of Tests.
207 - Duration in minutes all the .Stress Test Client. are to be run.
208 2. HostName
209 - Host name or IP address of the system where the CIM server is
210 expected to be running.
|
211 kumpf 1.3 3. Port.
|
212 j.alex 1.2 - Specifies Port number. The default port is 5988.
213 4. Number of Clients.
214 - Number of concurrent clients.
215 5. Name Space.
216 - Name space that will be utilized by the Stress Test Clients.
217 6. Class Name.
218 - Name of the existing class that will be utilized by the Stress
219 Test Clients.
220 7. SSL.
221 - Use SSL protocol between Stress Client and the CIM Server.
222 8. UserName.
|
223 kumpf 1.3 - User name that the Stress Test Clients will use to connect to
|
224 j.alex 1.2 the server.
225 9. Password.
226 - Password that is associated to the user name.
|
227 kumpf 1.3 10. ToleranceLevel.
|
228 j.alex 1.2 - This represents the percentage of the started "Stress Test
|
229 kumpf 1.3 client" that can be allowed to fail in every one minute
230 interval without resulting in the termination of the stress
|
231 j.alex 1.2 tests.
232
233 NOTE:
|
234 kumpf 1.3 Any reported client failure in a one minute interval corresponds
|
235 j.alex 1.2 to a "failure status" for that interval. The default is 0%,
|
236 kumpf 1.3 which means no failures are allowed, or any failures will result
|
237 j.alex 1.2 in termination of the stress test.
238
|
239 kumpf 1.3 11. ClientDuration.
|
240 j.alex 1.2 - Specifies duration in minutes for the corresponding stress
241 client before it should be stopped.
242 12. ClientWait.
|
243 kumpf 1.3 - Specifies waiting period in minutes before the corresponding
244 stress client is re-started again. "ClientDuration" when used
245 without "ClientWait" will not be re-run. "ClientWait" will
|
246 j.alex 1.2 only apply when used with "ClientDuration".
247 13. ClientName.
|
248 kumpf 1.3 - The name of the Client program that will be used by a
|
249 j.alex 1.2 TestWrapperStressClient instance.
250 14. Options.
|
251 kumpf 1.3 - The options that are desired for the corresponding Client
|
252 j.alex 1.2 program that will be used by a TestWrapperStressClient.
253
254 2.3.5 Instances of the Clients may also have unique configurable values.
255 Unique values can be specified for each instance in the
256 configuration file as shown in the Sample Configuration File below.
257 If any of the above parameters are repeated for a client instance,
258 then that particular instance(s) will utilize the new value.
259
260 2.3.6 Parameters "ClientName" and "Options" are only used for
|
261 kumpf 1.3 "TestWrapperStressClient".
|
262 j.alex 1.2
|
263 kumpf 1.3 2.3.7 The parameter "Hostname" will have to be the same for all the
|
264 j.alex 1.2 Clients.
|
265 kumpf 1.3
|
266 j.alex 1.2 NOTE:
267 - Options or values specific for each client or instance must be placed
|
268 kumpf 1.3 after the corresponding client or instance. Otherwise, the order of
269 the configuration variables in the configuration file is not
|
270 j.alex 1.2 important.
|
271 kumpf 1.3 - When using a Client program like a "CLI" or any other "clients"
272 with the "TestWrapperStressClient", ensure that options associated
|
273 j.alex 1.2 to the Client program will connect to the same CIM Server.
|
274 kumpf 1.3
|
275 j.alex 1.2 Example:
276 [clientname="CLI",options="niall -l <hostname>"]
277
|
278 kumpf 1.3 - Running without a failure implies that the client is communicating
279 with the CIMServer and has recieved responses other than an
|
280 j.alex 1.2 exception or a CIMError.
281
282 2.3.8 A Sample Configuration File:
283 ===========================
284
285 #Machine Host of Server to be tested
286 #If left blank then default is the local host.
287 Hostname=narten
288
289 # Total Duration of the Stress Client Test set as 300 minutes.
290 Duration=300
291
292 # Tolerance level in percentage
293 ToleranceLevel=0
294
295 # Number of Client instances
296 # The format for running a client program:
297 # <client program name>=<number of instances of clients>
298 # [option1=<data1>, option2=<data2>, option3=<data3>, ..... ]
299 #
300 # specify all the options for a Client instance or instances within
|
301 kumpf 1.3 # a square bracket "[ .... ]".
|
302 j.alex 1.2 #
303 #
304 # TestWrapperStressClient can only have the parameters "ClientName"
305 # and "Options". All other options will be ignored.
306 # [ClientName=<ClientProgram>,Options=<actual options for Client>]
307
308 ModelWalkStressClient=1
309 [NameSpace=/root/cimv2,ClassName=CIM_door,ClientDuration=60,ClientWait=10]
310 ModelWalkStressClient=5
311 WrapperStressClient=2
312 [ClientName=CLI,Options="-l narten niall"]
313 WrapperStressClient=10
314 [ClientName=CLI,Options="-l narten ei pg_computersystem"]
315 WrapperStressClient=5
316 [ClientName=osinfo,Options="-h narten",ClientDuration=120]
317
318 # Description:
319 # The above config values sets up the stress tests for the following
320 # stress test clients to be run concurrently:
|
321 kumpf 1.3 # 1. One ModelWalkStressClient which will use the Name space
322 # "/root/cimv2" and ClassName "CIM_door" overriding any
|
323 j.alex 1.2 # default value.
324 # This client will be run for 60 minutes then stopped for 10
325 # minutes before it is re-started again. This process will be
326 # repeated for the duration of the stress tests.
327 # 2. Five ModelWalkStressClient's using common values.
|
328 kumpf 1.3 # 3. Two WrapperStressClients will run the commands
329 # "CLI -l narten niall" concurrently.
|
330 j.alex 1.2 # 4. Ten WrapperStressCleints will concurrently run the command
|
331 kumpf 1.3 # "CLI -l narten ei pg_computersystem".
332 # 5. Five WrapperStressCleints will run the command "osinfo -h
|
333 j.alex 1.2 # narten" concurrently for 120 minutes and then stop.
334 #
335 # Note:
336 # The 17 wrapper clients instances are also connecting to the
|
337 kumpf 1.3 # same hostname as the other clients.(Please refer to items 3, 4
|
338 j.alex 1.2 # & 5 under Descriptions)
|
339 kumpf 1.3 # Please note that the "CLI" command uses -l option while the
340 # "osinfo" command uses -h option to set the hostname respectively.
|
341 j.alex 1.2
342 *******************************************************************************
343 3.0 Default Values:
344 ===============================================================================
345
|
346 kumpf 1.3 3.1 Default Hostname will be the local system where the stress tests are
|
347 j.alex 1.2 run.
348
349 3.2 Default port will be 5988.
350
351 3.3 Default "Duration" will be set as 180 minutes.
352
353 3.4 Default number of instances for each of the known clients is "5".
354
355 3.5 Default NameSpace when needed will be "root/cimv2".
356
357 3.6 Default UserName and Password will be blank.
358
359 3.7 Default will not use SSL.
360
361 3.8 Default percentage for "ToleranceLevel" will be set as 0%.
362
363 3.9 Default ClientWait & ClientDuration will be blank.
364
365 3.10 Default Clients for the stress tests will be set as follows:
|
366 kumpf 1.3
|
367 j.alex 1.2 ModelWalkStressClient=5
368 WrapperStressClient=5
369 [ClientName=CLI, Options=niall]
370
371 Note:
372 The above default sets up the stress tests for the following test
373 clients to be run concurrently:
374 Five ModelWalkStressClient instances using default values.
375 Five WrapperStressClient instances running the command "CLI niall".
376
377 *****************************************************************************
378 4.0 Running a Stress Test using default values:
379 =============================================================================
380 Assumptions:
381 CIMServer and Stress Test successfully build.
382 $PEGASUS_ROOT, $PEGASUS_HOME and other pegasus environments are
383 appropriately set.
384
385 4.1 Start the CIM Server on a specific hostname.
386
|
387 kumpf 1.3 4.2 Run the following command :
|
388 j.alex 1.2 (Ensure that the hostname specified below matches where the above
389 CIM Server resides.)
390
391 TestStressTestController -h <hostname>
|
392 kumpf 1.3
|
393 j.alex 1.2 NOTE:
394 If the <hostname> is local host run the command without the "-h" option
395 as follows:
396
397 TestStressTestController
398
399 4.3 The "stressTestController" will search for the default file
400 in the configuration directory.
401
|
402 kumpf 1.3 4.4 Assuming that we are using default values but connecting to the
403 specified host, the "stressController" will then invoke the Five
404 ModelWalkStressClient instances using default values & the Five
405 WrapperStressClient instances running the command
|
406 j.alex 1.2 "CLI niall -l <hostname>" .
407 (See section 3.0 (Default Values) above for reference)
408
409 4.5 The "TestStressTestController" will log status and other information in its
410 log file.
411
|
412 kumpf 1.3 4.6 Each instance of the invoked stressClients(ModelWalkStressClient &
|
413 j.alex 1.2 Wrapper StressClient) will connect to the specified hostname via the
414 default port and run its operations.
415
416 4.7 Note that the stress clients will repeat these operations until it
417 unexpectedly ends or is stopped by the "TestStressTestController".
418
419 4.8 TestStressTestController will monitor the overall status of these stress
420 clients for every One minute interval.
421 - If the overall stress client status is less than or equal to the
422 tolerance level then all is well and all the client instances are
423 allowed to chug along.
424 - if the overall stress client status is greater than the tolerance
425 level, then the stressTestController will terminate all the client
|
426 kumpf 1.3 instances and subsequently end the stress tests while reporting
|
427 j.alex 1.2 & loging this failure.
428
|
429 kumpf 1.3 4.9 If all is well then the tests will continue till the end of the
|
430 j.alex 1.2 duration of the stress tests, which for default will be 180 minutes.
431
432 *****************************************************************************
433 5.0 Running the new "stresstest" target:
434 =============================================================================
435
436 Assumptions:
437 CIMServer successfully build.
438 $PEGASUS_ROOT, $PEGASUS_HOME and other pegasus environments are
439 appropriately set.
440
441 5.1 Change directory to $PEGASUS_ROOT
442
443 5.2 Run the following command.
444 "make stresstests"
|
445 kumpf 1.3
446 - The new target "stresstests" will run the stress Tests
|
447 j.alex 1.2 ("stressTestController") using default configuration.
|
448 kumpf 1.3 The stress test programs are expected to be build already
|
449 j.alex 1.2 and ready to run.
450
451 - Any failure, will be reported accordingly and end the tests.
|
452 kumpf 1.3
|
453 j.alex 1.2 5.3 Target "tests" can also be run from $PEGASUS_ROOT/test/StressTests/tests
454 These tests will run some usage tests for the "TestStressTestController"
455 Any failure, will be reported accordingly and end the tests.
456 (These tests will be run as part of the nightly test.)
457
458 To invoke this target:
459 - Change directory to $PEGASUS_ROOT/test/StressTests/tests/
460 - Run the following command
461 "make tests"
462
463 *****************************************************************************
464 6.0 Stress Test Logs:
465 =============================================================================
466
467 Stress Test logs will contain general information about the stress tests.
468 Most of the informations concerns configuration properties, client status,
469 controller status, actual tolerance status, started/re-started clients,
470 stopped clients and other general information concerning the tests.
|
471 kumpf 1.3 A little more detailed logs and status reports are kept when the verbose
|
472 j.alex 1.2 option is selected.
473
474 6.1 Log files for the Stress Tests will be saved in the following directory:
475 $PEGASUS_HOME/test/StressTestController/log/
476
|
477 kumpf 1.3 6.2 Each test will generate two log files.(One for the controller and one
|
478 j.alex 1.2 for all the clients executed by this test.)
479
480 6.3 Log files for the controller can be found in above log directory:
481 and will be named as
482 "<controller_pid>_stressTest_<DDMMYYhhmmss>.log "
483 where,
484 "DD" denotes the day.
485 "MM" denotes the month.
486 "YY" denotes the year.
487 "hh" denotes the hour.
488 "mm" denotes the minutes.
489 "ss" denotes the seconds.
490
491 6.4 Log file for all the clients intiated by a controller can also be found
492 in the above log directory and will be named as
493 <controller-pid>_StressTestClients.log
|
494 kumpf 1.3
|
495 j.alex 1.2 where,
496 "controller-pid" denotes the pid of the TestStressTestController
497
498 *******************************************************************************
499 7.0 Stress Test Errors:
500 ===============================================================================
501 There are several sources of errors in the stress test.
502 They are
|
503 kumpf 1.3
|
504 j.alex 1.2 7.1 Stress Test Controller errors
505 (These failures will end the tests)
506 7.1.1 Invalid command options or operands.
507 7.1.2 Non-Existing Configuration file.
508 7.1.3 Syntax Error in Configuration file.
509 7.1.4 Invalid config property in config file.
510 7.1.5 Non-Existing Client PID file.(Failed to Communicate with Client)
511 7.1.6 Error reading Client PID File.
|
512 kumpf 1.3 7.1.7 Not within tolerance Error.
|
513 j.alex 1.2
514 7.2 Stress Test Client errors
|
515 kumpf 1.3 7.2.1 Time out waiting to connect
|
516 j.alex 1.2 This is an exception thrown by the client. It will be logged and
517 viewed by the controller as a failure.
518
|
519 kumpf 1.3 7.2.2 Time out waiting for response
|
520 j.alex 1.2 This is an exception thrown by the client. It will be logged and
521 viewed by the controller as a failure.
522
|
523 kumpf 1.3 7.2.2 Failed getInstance operation for some classes(See Bugs 4957 &
524 4268).
525
|
526 j.alex 1.2
527 7.3 Server Response Errors
|
528 kumpf 1.3
|
529 j.alex 1.2 7.3.1 Varied error response returned by the server.
|
530 kumpf 1.3 This kind of error will be logged and viewed by the controller as
|
531 j.alex 1.2 a failure.
532
533 *******************************************************************************
534 8.0 Other Considerations:
535 ===============================================================================
536 Compilation of the "Stress Test Clients" and the associated files will be
537 included in the nightly build. However, testing of the "Stress Test
538 Clients" and the associated files will NOT be included in the
539 nightly tests.
|