Quantcast

REST BC - character set or encoding

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

REST BC - character set or encoding

OSSTM
This post was updated on .
Hi,

I am working with the REST BC. In my BPEL I have an inbound REST Service with single Get operation. This in turn uses HTTP BC to get an XML from a webpage.

Then I return this XML to the REST BC and it returns the result, which can be XML or JSON.

However, the returned XML and JSON message is with a character encoding that corrupts non ANSI characters. I am unable to get it to return it in iso-8859-1 or utf-8 or something else that is readable.

Please note the XML which the HTTP BC returns does not have this problem.

I have been trying to manipulate the HTTP Headers but to no avail.

The REST BC works quite well, but this is a show stopper.

Does anyone know how to get around this?

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
Hi again,

I have investigated the issue and I am unable to get the REST BC to return a response with the iso-8859-1 character set.

Attached is a test project BPEL and CAP which shows this. It only includes a REST BC WSDL which returns a string literal assigned using the mapper. This string contains non-ASCII characters.
RESTTEST.zip

To run it I use curl in this way:

curl -i -H "accept-charset: ISO-8859-1" -H "accept:application/json"  http://localhost:9696/testcharset

This seems to be a bug which makes it unusable for me.

If you know of a way around this, please let me know.

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
Hi,

I have fixed the charset problem myself. I downloaded the REST BC source and found where the problem was. I built the REST BC and deployed to Glassfish and it works fine now.

If you would like to see how it was fixed don't hesitate to contact me. I don't know how to go about getting this checked into the source tree.

Others are bound to encounter this problem, so it would be good to get it fixed "at the source".

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: REST BC - character set or encoding

Frank Kieviet
Administrator
Hi,

If you're willing to do so, please contribute your changes back to the
community. There's a little bit of administravia involved, see
https://open-esb.dev.java.net/ContributionProcess.html, but once that's done
it should be very simple to add the change to the code base.

Thanks in advance!

Frank

Frank Kieviet
OpenESB Community Manager (http://open-esb.org)
Oracle, Monrovia, CA
Tel +1 626 471 6322
Blog: http://frankkieviet.blogspot.com/




> -----Original Message-----
> From: users-return-13527-frank.kieviet=[hidden email]
> [mailto:users-return-13527-frank.kieviet=[hidden email]] On
> Behalf Of OSSTM
> Sent: Tuesday, March 16, 2010 16:11
> To: [hidden email]
> Subject: Re: REST BC - character set or encoding
>
>
> Hi,
>
> I have fixed the charset problem myself. I downloaded the REST BC source
> and
> found where the problem was. I built the REST BC and deployed to Glassfish
> and it works fine now.
>
> If you would like to see how it was fixed don't hesitate to contact me. I
> don't know how to go about getting this checked into the source tree.
>
> Others are bound to encounter this problem, so it would be good to get it
> fixed "at the source".
>
> Thanks.
> --
> View this message in context: http://n2.nabble.com/REST-BC-character-set-
> or-encoding-tp4723479p4747078.html
> Sent from the OpenESB Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: REST BC - character set or encoding

OSSTM
No problem.

Will do that tomorrow (the time is 23:19 over here now).

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: REST BC - character set or encoding

OSSTM
Hi,

I did not commit the fix since there seems to be a wider problem.

My fix works in some cases and not in others. Using Content-Type of "UTF-8" never seems to work, but "ISO-8859-1" sometimes works but not consistently.

It seems that somewhere along the path, REST BC corrupts all non-ASCII characters. The exact same BPEL using SOAP BC instead of REST BC does not have this problem.

There are also other related bugs, for example, when the Accept header is : "text/html,application/xml" the REST BC returns a response with Content-Type "application/json" but the content nevertheless is XML.

Removing the "text/html" from the Accept header makes it work ok.

Is there nobody using the REST BC in this community?

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: REST BC - character set or encoding

OSSTM
Hi again,

My previous fix did work when the response included content with the ISO-8859-1 charset but failed when I tried it with the UTF-8 charset, which is what my client requires. It was a mystery to me why ISO-8859-1 worked but not UTF-8.

I dug into the code once again and I think I have found the culprit. It was much simpler than the fix I attempted earlier which involved working with the HTTP Headers.

The problem lies in the ConvertXmlToString method in the JBIMessageUtil class, a utility method used throughout the code.

Here is the code:

 public static String convertXmlToString(Source source) throws Exception {
       
        ByteArrayOutputStream baos = new ByteArrayOutputStream();

        Transformer transformer = TransformerFactory.newInstance().newTransformer();
        transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");


        transformer.transform(source, new StreamResult(baos));
     
        return baos.toString(); //problem lies here
    }


Now, the ByteArrayOutputStream.toString() method uses the platform's default encoding to transform the ByteArray into a String.

I changed that to toString("UTF-8") ( that is: "return baos.toString("UTF-8")"; ) and the problem appears to have been fixed when UTF-8 is what you want, although I can't confirm that until after I test it with real world use cases, which I will do after Easter holiday when I return to work.

The reason ISO-8859-1 charset worked with my previous fix is probably because that is my platform default encoding. After this latter fix you always get UTF-8, no matter what you request. So, now, if you set the request header to include "accept:application/xml,charset=iso-8859-1", you get corrupted text.

A proper fix would make the request header's charset propagate to the convertXmlToString() method, which would then encode the text with that charset, or if no particular charset is requested, it should use the platform default charset.

I'd rather not invest my time into going down that route until I get some feedback from you (the community or Oracle) about this.

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: REST BC - character set or encoding

OSSTM
There seems to be a similar method in the HTTP BC code:

Util.java, package com.sun.jbi.httpsoapbc.util:
public static String toXml(Node node, String encoding, boolean omitXMLDeclaration) throws Exception {
        String ret = null;
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        Transformer trans = cTransformerPool.retrieve();
        try {
            trans.setOutputProperty(OutputKeys.ENCODING, encoding);
            trans.setOutputProperty(OutputKeys.INDENT, "yes");
            trans.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "4");
            trans.setOutputProperty(OutputKeys.METHOD, "xml");
            trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, omitXMLDeclaration? "yes":"no");
            trans.transform(new DOMSource(node), new StreamResult(baos));
            ret = baos.toString(encoding);
        } finally {
            cTransformerPool.relinquish(trans);
        }



JbiHelper.java, package test.jbi.integration.testse.impl:
        public static String transformToString(Source src, String encoding)
                        throws Exception {
                Transformer trans = sTransformerFactory.newTransformer();
                String xmlData = null;
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                trans.setOutputProperties(null); // reset anything previously set
                trans.setOutputProperty(OutputKeys.ENCODING, encoding);
                trans.setOutputProperty(OutputKeys.INDENT, "yes");
                trans.setOutputProperty(OutputKeys.METHOD, "xml");
                trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
                trans.transform(src, new StreamResult(baos));
                xmlData = baos.toString(encoding);
                return xmlData;
        }


As you can see, the encoding is used in these methods, which otherwise seem to be doing similar things.

Best regards.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

Marcelo C. Fernandes
Hi OSSTM,

I have the same problem you had.
Did you commited the source code?

Do you mind sending me you the binary file of REST BC with your fix so that I can make sure it solve the problema my application has?

---
Marcelo Carvalho Fernandes
Arquitetura de TI
Infoglobo Comunicação e Participações S.A.
+55 21 2534-9718


2010/4/1 OSSTM <[hidden email]>

There seems to be a similar method in the HTTP BC code:

Util.java, package com.sun.jbi.httpsoapbc.util:
public static String toXml(Node node, String encoding, boolean
omitXMLDeclaration) throws Exception {
       String ret = null;
       ByteArrayOutputStream baos = new ByteArrayOutputStream();
       Transformer trans = cTransformerPool.retrieve();
       try {
           trans.setOutputProperty(OutputKeys.ENCODING, encoding);
           trans.setOutputProperty(OutputKeys.INDENT, "yes");

trans.setOutputProperty("{<a href="http://xml.apache.org/xslt}indent-amount" target="_blank">http://xml.apache.org/xslt}indent-amount", "4");
           trans.setOutputProperty(OutputKeys.METHOD, "xml");
           trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION,
omitXMLDeclaration? "yes":"no");
           trans.transform(new DOMSource(node), new StreamResult(baos));
           ret = baos.toString(encoding);
       } finally {
           cTransformerPool.relinquish(trans);
       }


JbiHelper.java, package test.jbi.integration.testse.impl:
       public static String transformToString(Source src, String encoding)
                       throws Exception {
               Transformer trans = sTransformerFactory.newTransformer();
               String xmlData = null;
               ByteArrayOutputStream baos = new ByteArrayOutputStream();
               trans.setOutputProperties(null); // reset anything previously set
               trans.setOutputProperty(OutputKeys.ENCODING, encoding);
               trans.setOutputProperty(OutputKeys.INDENT, "yes");
               trans.setOutputProperty(OutputKeys.METHOD, "xml");
               trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
               trans.transform(src, new StreamResult(baos));
               xmlData = baos.toString(encoding);
               return xmlData;
       }

As you can see, the encoding is used in these methods, which otherwise seem
to be doing similar things.

Best regards.
--
View this message in context: http://n2.nabble.com/REST-BC-character-set-or-encoding-tp4723479p4840056.html
Sent from the OpenESB Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


- - - - - - - - - - - - - - - - - - - - AVISO IMPORTANTE / IMPORTANT NOTICE - - - - - - - - - - - - - - - - - - - - - -

Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode 
utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem 
tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada 
de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir 
ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento. 



This message may include confidential information and only the intended addressee have the right to use it as is, or 
any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or 
erroneous transmission, please notify the senderdelete it from your system immediately. This communication environment 
is controlled and monitored. 



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	
	
	
	
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

Marcelo C. Fernandes
Hi OSSTM,

I have the same problem. Did you commited the source code already?

Do you mind sending me your fixed version of REST BC to see if it works for me? The binary file would be enough.

Thanks in advance,

---
Marcelo Carvalho Fernandes
Arquitetura de TI
Infoglobo Comunicação e Participações S.A.
+55 21 2534-9718
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
In reply to this post by Marcelo C. Fernandes
Hi Marcelo,

I did fix it but only with hard-coding the UTF-8 encoding into the relevant source code. That was enough for my client. But I know how to fix it properly and it should be easy.

If the hard-coded quick fix is enough for you, I can send you the binary, sure.

Let me know,

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

Marcelo C. Fernandes
Yes! That's enough! 

Thanks in advance,


---
Marcelo Carvalho Fernandes
Arquitetura de TI
Infoglobo Comunicação e Participações S.A.
+55 21 2534-9718


2010/10/29 OSSTM <[hidden email]>

Hi Marcelo,

I did fix it but only with hard-coding the UTF-8 encoding into the relevant
source code. That was enough for my client. But I know how to fix it
properly and it should be easy.

If the hard-coded quick fix is enough for you, I can send you the binary,
sure.

Let me know,

Thanks.
--
View this message in context: http://openesb-users.794670.n2.nabble.com/REST-BC-character-set-or-encoding-tp4723479p5686128.html
Sent from the OpenESB Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


- - - - - - - - - - - - - - - - - - - - AVISO IMPORTANTE / IMPORTANT NOTICE - - - - - - - - - - - - - - - - - - - - - -

Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode 
utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem 
tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada 
de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir 
ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento. 



This message may include confidential information and only the intended addressee have the right to use it as is, or 
any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or 
erroneous transmission, please notify the senderdelete it from your system immediately. This communication environment 
is controlled and monitored. 



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	
	
	
	
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
Hi,

is it working for you?

Best regards,
OSSTM
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

Marcelo C. Fernandes
hallgrimurb,

I've just discovered that it was the "Do Marshal" task that was converting some characters in my XML message and not Rest BC.
Therefore, it was not necessary to deploy your REST BC snapshot.

I'll create another thread to address the issue with the Do Marshal. Does it have any value to you if I test/validate the modification you made in Rest BC?


---
Marcelo Carvalho Fernandes
Arquitetura de TI
Infoglobo Comunicação e Participações S.A.
+55 21 2534-9718


2010/11/3 OSSTM <[hidden email]>

Hi,

is it working for you?

Best regards,
OSSTM
--
View this message in context: http://openesb-users.794670.n2.nabble.com/REST-BC-character-set-or-encoding-tp4723479p5702047.html
Sent from the OpenESB Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


- - - - - - - - - - - - - - - - - - - - AVISO IMPORTANTE / IMPORTANT NOTICE - - - - - - - - - - - - - - - - - - - - - -

Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode 
utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem 
tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada 
de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir 
ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento. 



This message may include confidential information and only the intended addressee have the right to use it as is, or 
any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or 
erroneous transmission, please notify the senderdelete it from your system immediately. This communication environment 
is controlled and monitored. 



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	
	
	
	
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

OSSTM
Hi,

No need to test it, I know it works.

Best regards,
OSSTM
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: REST BC - character set or encoding

Marcelo C. Fernandes
Ok OSSTM.

Thank you very much for sending me the file.
I'll keep it because we never know if it will be required in the future :-)

---
Marcelo Carvalho Fernandes
Arquitetura de TI
Infoglobo Comunicação e Participações S.A.
+55 21 2534-9718


2010/11/4 OSSTM <[hidden email]>

Hi,

No need to test it, I know it works.

Best regards,
OSSTM
--
View this message in context: http://openesb-users.794670.n2.nabble.com/REST-BC-character-set-or-encoding-tp4723479p5705014.html
Sent from the OpenESB Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


- - - - - - - - - - - - - - - - - - - - AVISO IMPORTANTE / IMPORTANT NOTICE - - - - - - - - - - - - - - - - - - - - - -

Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode 
utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem 
tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada 
de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir 
ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento. 



This message may include confidential information and only the intended addressee have the right to use it as is, or 
any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or 
erroneous transmission, please notify the senderdelete it from your system immediately. This communication environment 
is controlled and monitored. 



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	
	
	
	
Loading...