pegasus/doc/Globalization_HOWTO.htm - diff

Return to Globalization_HOWTO.htm CVS log

Up to [Pegasus] / pegasus / doc

Diff for /pegasus/doc/Globalization_HOWTO.htm between version 1.5 and 1.5.2.1

version 1.5, 2003/10/13 21:18:22

version 1.5.2.1, 2003/12/01 19:57:38

Line 8

</head>

<body text="#000000" bgcolor="#ffffff" link="#0000ef" vlink="#55188a"

alink="#ff0000">

<center>Globalization HOWTO

<big><big><big>Globalization HOWTO</big></big></big>

Release: Pegasus 2.3

Author: Chuck Carmack (carmack@us.ibm.com)

July 28, 2003

December 1, 2003

</center>

Change History:

<table cellpadding="2" cellspacing="2" border="1"

style="text-align: left; width: 100%; margin-left: auto; margin-right: auto;">

<tbody>

<tr>

<td style="vertical-align: top;">01/12/03

</td>

<td style="vertical-align: top;">carmack

</td>

<td style="vertical-align: top;">Section 2.2.2.  Changed how

the package name parameter should be used.  It should no longer be

used as part of the table name inside the bundle.

</td>

</tr>

</tbody>

</table>

<h2> 1.0 Introduction</h2>

Line 25

Line 44

locale-neutral.  In other words, the program should be able to run

in any locale without change.  There are several categories in a

locale, including the language of message strings, date format, time

format, etc.  For release 2.3, the Pegasus server is concerned

format, etc.  For release 2.3, the Pegasus server is concerned with

with the language of the message strings it returns to its clients.

the language of the message strings it returns to its clients.

To support internationalization, a program is designed to do the

following:

Line 39

Line 58

and usually has converters on the platform, it is a good choice for the

'normalized' internal character set.    The most

'interoperable' solution for external data is to support UTF-8 (eg.

network and file system data).  The internal data is usually

network and file system data).  The internal data is usually UTF-16

UTF-16 (or UCS-2, but that is deprecated).</li>

(or UCS-2, but that is deprecated).</li>

<li> Extract locale-sensitive resources, such as message

strings, from the code to external resource files.  Typically, the

Line 79

Line 98

files.</li>

<li> APIs were added for clients to associate a language with

the CIM objects they are sending to Pegasus.  Also, APIs were

the CIM objects they are sending to Pegasus.  Also, APIs were added

added for clients to determine the language of the error message or CIM

for clients to determine the language of the error message or CIM

object that Pegasus returns.</li>

<li> APIs were added for providers to determine the language of

CIM objects sent by the client.  Also, APIs were added for

providers to associate a language with the CIM object, or error

providers to associate a language with the CIM object, or error message,

message, they return to the client.</li>

they return to the client.</li>

</ul>

Please refer to PEPs 56 and 58 for details about the globalization

Line 137

Line 156

<li> The Pegasus 2.2 methods that take a char *, or return char *, are

unchanged.  Code written to Pegasus 2.2 may have expected to store

8-bit ASCII (ISO-8859-1) characters into String.  These methods

will convert the input to UTF-16 from 8-bit ASCII.  (This is

will convert the input to UTF-16 from 8-bit ASCII.  (This is simple

simple because UTF-16 is a superset of 8-bit ASCII - simply need to

because UTF-16 is a superset of 8-bit ASCII - simply need to prepend

prepend '\0' to each char).  The Pegasus 2.2 methods that return

'\0' to each char).  The Pegasus 2.2 methods that return char data

char data will attempt to convert from the UTF-16 internal

will attempt to convert from the UTF-16 internal representation to

representation to 8-bit ASCII.  Characters that cannot be

8-bit ASCII.  Characters that cannot be converted will be replaced

converted will be replaced with a substitution character.</li>

with a substitution character.</li>

<li> All methods that take or return Char16 data are

unchanged.  The String class now supports UTF-16 data in Char16,

although surrogate pairs will require two consecutive Char16

objects.  The String class does NO checking for unmatched

objects.  The String class does NO checking for unmatched surrogate

surrogate pairs.</li>

pairs.</li>

<li> New methods have been added to take and return UTF-8

data.  The String class will convert between UTF-8 and the UTF-16

internal representation as needed.  These new methods will use

internal representation as needed.  These new methods will use char

char * parameters, but will be clearly labelled as UTF-8 methods.</li>

* parameters, but will be clearly labelled as UTF-8 methods.</li>

</ul>

Line 162

Line 181

superset of 8-bit ASCII.  Any String object containing EBCDIC data

will not work if it is used by Pegasus to read or write data from

external sources, such as the network or repository files.  In

other words, any String containing EBCDIC data should not leave the

other words, any String containing EBCDIC data should not leave the code

code using it.

using it.

<h3> 2.2 Localization Support</h3>

Line 202

Line 221

CIM clients may use the Accept-Language HTTP header to specify the

languages they wish to be returned in the CIM response message. 

CIM clients may also use the Content-Language header to tag the

CIM clients may also use the Content-Language header to tag the language

language of any CIM objects they are sending to the server in the CIM

of any CIM objects they are sending to the server in the CIM request

request message.  The server, and providers, should attempt to

message.  The server, and providers, should attempt to return

return error messages and CIM objects in one of the accept languages

error messages and CIM objects in one of the accept languages requested

requested by the client.  The server and providers should set the

by the client.  The server and providers should set the

Content-Language header in the CIM response message to indicate which

Content-Language header in the CIM response message to indicate which of

of the requested languages they are returning.

the requested languages they are returning.

NOTE:  Localization support was not added for the MOF files and

repository in Pegasus 2.3.  The #pragma locale, #pragma

Line 217

Line 236

classes, qualifiers, and instances stored in the repository are not

tagged with a language.  The Accept-Language and Content-Language

headers will be ignored for repository operations.  However, since

the repository will support UTF-8,  characters for any language

the repository will support UTF-8,  characters for any language may

may be stored there.

be stored there.

NOTE:  Since the Content-Language header applies to the entire

HTTP message, it applies to the entire CIM-XML document.  This

Line 227

Line 246

remain until the CIM standard has been updated to support language tags

tied to individual CIM values.  From the client perspective, it is

possible for Pegasus to send a CIM response with NO Content-Language,

even if the client had sent Accept-Language.   This can

even if the client had sent Accept-Language.   This can happen

happen if Pegasus does not know the language of the response.  An

if Pegasus does not know the language of the response.  An example

example is a request that was sent to a Pegasus 2.2 provider. 

is a request that was sent to a Pegasus 2.2 provider.  Another

Another example is an enumerated response where each provider returned

example is an enumerated response where each provider returned a

a different language.  Please refer to PEP58 for details on these

different language.  Please refer to PEP58 for details on these

provider scenarios.

Pegasus 2.3 has added classes for the localization support.

Line 290

Line 309

href="http://oss.software.ibm.com/icu/userguide/ResourceManagement.html">Resource

Management</a>  section of the <a

href="http://oss.software.ibm.com/icu/userguide/">ICU User Guide</a>

.  This section will tell you how to

.  This section will tell you how to create and organize your

create and organize your resource bundles for different

resource bundles for different languages.  Note:  your

languages.  Note:  your resource bundles should be organized

resource bundles should be organized in a tree structure similiar to

in a tree structure similiar to the one shown in the Resource

the one shown in the Resource Management section, including the empty

Management section, including the empty bundles in the tree.  It

bundles in the tree.

is recommended that you ship a root resource bundle to be used as the

fallback in case the client requests a language that you are not

It is recommended that you ship a root resource bundle to be used as

the fallback in case the client requests a language that you are not

supporting.  The Pegasus make files are set up to automatically

create and compile a root resource bundle for you.  For Pegasus

2.3, the make will use your "en" bundle, upper case all the messages,

and then put the uppercased messages into the root bundle.  The

uppercasing of the messages is necessary to create a "fallback" root

bundle that contains invariant characters across all EBCIC and

bundle that contains invariant characters across all EBCDIC and

ASCII codepages.

NOTE:  When creating your resource bundles, the name of the

table resource should contain the package name.    For

table resource should not

example, if you have a bundle with a package name of "xyz", then the

contain the package name.    For example, if you

"en" bundle should start like this:

have a bundle with a package name of "xyz", then the "en" bundle should

xyz_en:table {

start like this:

en:table {

..... messages here

}

}

not like this:

en:table {

xyz_en:table {

..... messages here

}

}

This is different than some of the examples in the <a

href="http://oss.software.ibm.com/icu/userguide/ResourceManagement.html">Resource

Management</a> section, but is needed because the -p option is not used

This is needed because the package name (-p) option is used by the

on genrb by the make files.

Pegasus make files on the call to genrb.

NOTE:  Pegasus 2.3 only supports simple string resources in the

ICU resource bundles.  String resources may only be loaded by

Line 360

Line 382

Code that needs to load a message in Pegasus does not call ICU

directly.  Two message loading classes were added for Pegasus

2.3:  MessageLoader and MessageLoaderParms.  These classes

2.3:  MessageLoader and MessageLoaderParms.  These classes are

are abstractions designed to hide of the actual loader used (but note

abstractions designed to hide of the actual loader used (but note that

that at the time of writing, only ICU is supported).   The

at the time of writing, only ICU is supported).   The

MessageLoader is used to load a message using a list of preferrred

languages.  The parameters to MessageLoader are encapsulated in a

MessageLoaderParms object.

Line 371

Line 393

Content-Language header, and the ICU resource bundles, join up. 

The MessageLoader class is designed to receive an AcceptLanguages

object, and a set of parameters indicating the bundle base-name and

message ID to use.  The AcceptLanguages object contains the list

message ID to use.  The AcceptLanguages object contains the list of

of requested languages sent by the client.  The MessageLoader

requested languages sent by the client.  The MessageLoader

searches for the message in the set of bundles named with the

searches for the message in the set of bundles named with the base-name,

base-name, using the AcceptLanguages for the list of specific

using the AcceptLanguages for the list of specific translated bundles

translated bundles to search.  The MessageLoader returns the

to search.  The MessageLoader returns the message that it found,

message that it found, along with a ContentLanguages object indicating

along with a ContentLanguages object indicating the language of the

the language of the message.  The ContentLanguages object should

message.  The ContentLanguages object should be used to indicate

be used to indicate the language of the response sent back to the

the language of the response sent back to the client.

client.

The MessageLoaderParms object contains the parameters to load the

message.  There are many parameters, but many can be allowed to

Line 411

Line 432

<td>Input.

Optional

Default: $PEGASUS_HOME/msg/pegasus/pegasusServer</td>

<td>Path to the root resource bundle file which contains the

<td>Path to the resource bundle file which contains the

msg_id.

Note: Only specify the path down to the bundle base-name.  Do not

append a language tag, such as "_root" or "_en".  Do not append a

Line 426

Line 447

Default: AcceptLanguages::EMPTY</td>

<td>Contains the list of preferred languages, in priority

order.  This is combined with msg_src_path to determine which

resource bundles to search for for the msg_id.   If not

resource bundles to search for for the msg_id.   If not empty,

empty, overrides useThreadLocale and useProcessLocale.</td>

overrides useThreadLocale and useProcessLocale.</td>

</tr>

<tr>

<td>ContentLanguages contentlanguages;</td>

Line 461

Line 482

resource bundles if the msg_id cannot be found.  Note: the

recommended setting is false if you are using an AcceptLanguages from a

CIM client.  The Accept-Languages HTTP header from the client

contains the fallback specifications.</td>

contains the fallback specifications.  Using ICU's fallback in this

case may lead to returning a language that the client didn't ask for.</td>

</tr>

<tr>

<td>Formatter::Arg arg0;

Line 490

Line 512

Pegasus Thread in which the caller's code is running.  This

AcceptLanguages object reflects the languages requested by the

client.  This is useful for code that may not have access to the

AcceptLanguages from the client.  Pegasus sets this

AcceptLanguages from the client.  Pegasus sets this AcceptLanguages

AcceptLanguages object into the Thread of providers and internal

object into the Thread of providers and internal Pegasus code. 

Pegasus code.  For this reason, it is recommended that provider

For this reason, it is recommended that provider and internal Pegasus

and internal Pegasus code use the "useThreadLocale" flag instead of

code use the "useThreadLocale" flag instead of explicity passing in an

explicity passing in an AcceptLanguages object.  See the Provider

AcceptLanguages object.  See the Provider Developer and Pegasus

Developer and Pegasus Developer sections for details.

Developer sections for details.

The "useProcessLocale" flag can be used to tell MessageLoader to use

the default locale of the process, as determined by ICU.  This is

useful for situations where the caller is not localizing for a client

request.  The caller may itself be a client (eg. cimconfig), or

request.  The caller may itself be a client (eg. cimconfig), or may

may need to log messages to the system log in the locale of the Pegasus

need to log messages to the system log in the locale of the Pegasus

server process.  See the CLI Messages and Logger Messages sections

below.

Line 523

Line 545

the "fallback" mechanism described in the ICU Resource Management

section.  This is because the Accept-Language header itself

describes the fallback that the client wants.  However, the

MessageLoader does "fallback" to the root resource bundle if none of

MessageLoader does "fallback" to the root resource bundle if none of the

the languages in AcceptLanguages can be found.  If the root

languages in AcceptLanguages can be found.  If the root resource

resource bundle cannot be found, then the default_msg is

bundle cannot be found, then the default_msg is returned.  The

returned.  The "useICUFallback" flag can be set to have

"useICUFallback" flag can be set to have MessageLoader use ICU fallback

MessageLoader use ICU fallback on all message load attempts. 

on all message load attempts.  However, usage of this flag for

However, usage of this flag for client requests may lead to incorrect

client requests may lead to incorrect results.  For example, a

results.  For example, a client sets Accept-Language to french,

client sets Accept-Language to french, german, and spanish, in that

german, and spanish, in that order, but there is no french resource

order, but there is no french resource bundle.  A call to

bundle.  A call to MessageLoader with useICUfallback == true would

MessageLoader with useICUfallback == true would cause the root resource

cause the root resource bundle string to be returned on the attempt to

bundle string to be returned on the attempt to load from the french

load from the french bundle.  But the client requested german to

bundle.  But the client requested german to be the fallback after

be the fallback after french.

french.

Please refer to the following files for details on the new Pegasus

classes.

Line 569

Line 591

<ul>

<li> If you want to claim that you are globalized, no hardcoded

messages!</li>

<li> Avoid combining messages in the code from other messages.

<li> Avoid creating a message in the code by combining other

When you do this you are assuming that you know the grammar for every

messages.  When you do this you are assuming that you know the

language.</li>

grammar for every language.</li>

<li> String substitutions into messages are generally untranslated,

ie. not loaded from the resource bundle.   Example: a file

name.</li>

Line 587

Line 609

support localization.  Constructors have been added that take a

MessageLoaderParms object.  These constructors will use the

MessageLoaderParms object to call the MessageLoader to load the

localized exception message.  The localized message is saved in

localized exception message.  The localized message is saved in the

the Exception.  The ContentLanguages object returned by

Exception.  The ContentLanguages object returned by MessageLoader

MessageLoader is also saved in the Exception.  This indicates the

is also saved in the Exception.  This indicates the language of

language of the message.  The ContentLanguages object is used

the message.  The ContentLanguages object is used later to set the

later to set the Content-Language header in the HTTP message to the

Content-Language header in the HTTP message to the client.

client.

The old Exception constructors that take a String will remain.

These should be used in cases where the code throwing the exception is

Line 614

Line 635

<ul>

<li> Are there localized string properties that need to be

supported?  If so, then the client will use Accept-Language to

request specific languages for these properties.  If the

request specific languages for these properties.  If the properties

properties are read-only, use MessageLoader to load the localized

are read-only, use MessageLoader to load the localized strings for the

strings for the properties.</li>

properties.</li>

<li> If you have a localized read/write string property, then the

client will use Content-Language to set the property with an associated

language.  The client will expect to be able to retrieve the

Line 645

Line 666

<ul>

<li> Pegasus will set the Accept-Language from the client into the

thread in which the provider is running.  By using the

useThreadLocale setting in MessageLoaderParms, providers can easily

useThreadLocale setting in MessageLoaderParms, providers can easily load

load strings using the client's requested Accept-Language.  The

strings using the client's requested Accept-Language.  The

provider does not need to know what the Accept-Language is.  This

is the recommended method to load messages based on the client's

is the recommended method to load messages based on the client's request.</li>

request.</li>

<li> The OperationContext will contain an AcceptLanguages object

that has the Accept-Language requested by the client.  The

that has the Accept-Language requested by the client.  The provider

provider can use this AcceptLanguages object to load strings with

can use this AcceptLanguages object to load strings with MessageLoader.</li>

MessageLoader.</li>

</ul>

The OperationContext will also contain a ContentLanguages object that

Line 669

Line 688

returning by calling setContext( ) on the ResponseHandler.  This

will be used to set the Content-Language in the CIM response message

sent back to the client.  If setContext( ) is not called, then no

Content-Language will be returned to the client.  setContext( )

Content-Language will be returned to the client.  The setContext( )

should only be called once per response.

function should only be called once per response.

<h3> 3.2 Sample Code</h3>

Line 819

Line 838

What happens if the client sets the read/write property with a

Content-Language that is not one of the supported languages for the

read/only property?  This provider allows the client to set any

language into the read/write property, and get that property back in

language into the read/write property, and get that property back in the

the same language.  This becomes an issue when the client does a

same language.  This becomes an issue when the client does a

getInstance( ) later, because the Content-Language on the returned

instance applies to all the properties.  A related issue is what

instance applies to all the properties.  A related issue is what to

to return for Content-Language when the client does enumerateInstances,

return for Content-Language when the client does enumerateInstances,

but the instances have different languages.  Recall that

Content-Language applies to the entire response (a limitation in the

Content-Language applies to the entire response (a limitation in the CIM

CIM specification).

specification).

NOTE:  Indication Providers have other special considerations

for language support.  Please refer to  PEP58.

Line 852

Line 871

(SC41-5607-02).  The Pegasus string literals will be compiled with

the UTF-8 compile switch described in this section.  OS/400

provider developers should strongly consider using the same compile

switch for their string literals.  This would allow the literals

switch for their string literals.  This would allow the literals to

to match the UTF-8 encoding expected by the C-runtime.</li>

match the UTF-8 encoding expected by the C-runtime.</li>

</ul>

<h2> 4. 0 Client Developers</h2>

Line 971

Line 990

The make messages target will compile these resource bundles.

Note:  As described above, the resource bundle path in

MessageLoaderParms defaults to the server resource bundle.  For

MessageLoaderParms defaults to the server resource bundle.  For CLI

CLI messages, you will need to specify the bundle for your CLI.

messages, you will need to specify the bundle for your CLI.

<h3> 5.2 Server Messages</h3>

Line 990

Line 1009

AcceptLanguages object and a ContentLanguages object.  For

CIMRequestMessage, these objects contain the Accept-Language and

Content-Language headers that were built from the client request. 

For CIMResponseMessage, the ContentLanguages object is used to build

For CIMResponseMessage, the ContentLanguages object is used to build the

the Content-Language header associated with the CIM objects in

Content-Language header associated with the CIM objects in the

the response message.  The AcceptLanguages object in the

response message.  The AcceptLanguages object in the

CIMResponseMessage is ignored.

The localization of the cimException object in the

CIMResponseMessage is handled separately from the CIM objects. 

CIMResponseMessage is handled separately from the CIM objects.  The

The message string in the cimException object is assumed to have been

message string in the cimException object is assumed to have been

localized by the time it is built into the XML.  For this reason,

the localization of the exception is the responsibility of the code

throwing the exception.  (The goal of the design is to make that

Line 1019

Line 1038

AcceptLanguages into its Thread from the AcceptLanguages in the

CIMRequestMessage object that it dequeues.  This sets the global

langauge for all the functions in the same thread that are called below

handleEnqueue.  If you are writing a new service that

handleEnqueue.  If you are writing a new service that processes

processes requests, or discover a request service that was missed,

requests, or discover a request service that was missed, please do

please do this.  The CIMOperationRequestDispatcher service is

this.  The CIMOperationRequestDispatcher service is an example.

an example.

How to Throw a Localized Exception from Server code:

Line 1075

Line 1093

client.

For Pegasus 2.3, these types of macro calls can stay.  The

TraceableCIMException constructed by the macro will

TraceableCIMException constructed by the macro will "re-localize". 

"re-localize".  That is, the "CIM" part of the message (the part

That is, the "CIM" part of the message (the part based on the error

based on the error code) will be localized at throw time, and the

code) will be localized at throw time, and the ContentLanguages

ContentLanguages re-established.  A key is to avoid a "language

re-established.  A key is to avoid a "language mismatch" problem

mismatch" problem between the CIM part of the message and the extra

between the CIM part of the message and the extra part of the

part of the message.  The design point here is that all internal

message.  The design point here is that all internal exceptions

exceptions thrown by Pegasus code are localized using the global

thrown by Pegasus code are localized using the global AcceptLanguages

AcceptLanguages of the Thread...see above.

of the Thread...see above.

In the future, it will be safer and more maintainable to use of

the  new "localized" flavors of the macro.  For example:

Line 1119

Line 1137

New methods have been added to Logger to take a message ID of a message

to be loaded from the Pegasus server resource bundle.  The caller

is only required to pass in the message ID, the old "hardcoded"

is only required to pass in the message ID, the old "hardcoded" message,

message, and the args.  The Logger will use MessageLoader to load

and the args.  The Logger will use MessageLoader to load the

the message in the locale of the Pegasus server process, using

message in the locale of the Pegasus server process, using the

the hardcoded message as the default string.  Please refer to

hardcoded message as the default string.  Please refer to

pegasus/src/Pegasus/Logger.h.

Note:  Messages sent to the "logs", whether the system logs or

the Pegasus log file, are converted to UTF-8 before being sent.

Line 1134

Line 1152

-- the user should not be required to tell the CLI what the locale

is.   For the CLIs that are CIM clients (cimconfing,

cimprovider) there are two sets of messages to localize  --

messages generated in the CLI process itself, and messages returned

messages generated in the CLI process itself, and messages returned from

from the Pegasus server .  For CLIs that are directly linked into

the Pegasus server .  For CLIs that are directly linked into

Pegasus (cimmofl), all the messages are generated in the CLI's process,

but the CLI may call Pegasus APIs that are coded to localize based on a

client's requested languages.

Line 1143

Line 1161

Code in the client side of the client/server CLIs (eg. cimconfig,

cimmof), or in directly linked CLIs (cimmofl), should use the

_useProcessLocale "master switch" described in the Message Loading

section.  This will cause all messages, including exceptions

section.  This will cause all messages, including exceptions thrown

thrown by Pegasus APIs,  to be loaded in the locale based on the

by Pegasus APIs,  to be loaded in the locale based on the

environment in which the program is running.  This locale can be

set by the user before running the program.

Line 1164

Line 1182

obtaining a copy  of this software and associated documentation

files (the "Software"), to deal in the Software without restriction,

including without limitation the rights to use, copy, modify, merge,

publish, distribute, sublicense, and/or sell copies of the Software,

publish, distribute, sublicense, and/or sell copies of the Software, and

and to permit persons to whom the Software is furnished to do so,

to permit persons to whom the Software is furnished to do so, subject

subject to the following conditions:

to the following conditions:

THE ABOVE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE SHALL BE

INCLUDED IN ALL COPIES OR SUBSTANTIAL PORTIONS OF THE SOFTWARE. THE

SOFTWARE IS PROVIDED  "AS IS", WITHOUT WARRANTY OF ANY KIND,

Legend:

Removed from v.1.5
changed lines
	Added in v.1.5.2.1

No CVS admin address has been configured