Non-Responsive Glassfish/JBI Bridge after some use

classic Classic list List threaded Threaded
13 messages Options
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
We have a basic use case where we exposed couple of web services using CASA, A BPEL module and an EJB Module. All these are bundled of course and deployed together. We create EJBs and then expose those as web services using the netbeans feature of creating web service from WSDL option. We create an abstract WSDL and then use that to create a web service as a Stateless Session EJB. Netbeans successfuly does everything and our EJB is accessible over 8080 port too (default http port of glassfish).

Issue is that after a use of 5-6 days where maximum 500-600 requests are served by our BPEL process, Glassfish becomes non-responsive. Strange thing is that EJB part is not accessible to BPEL and it throws no error too. We can see the MessageExchange being created where Java-ee-engine is the provider and java-bpel-engine is the consumer. The message goes and never is picked up by the java-ee-engine. When we call the EJB directly over 8080, everything works fine but the BPEL call is not served by the BPEL.

We are running OpenESB version 2.3 which comes with Glassfish 2.1.1. 3000m is the min and max heap memory allocated to the JVM in which these two are running without any other addition to the default configuration of the domain.xml. We suspect the issue with the JBI Bridge but are unable to find the answer.

Our OS is windows server 2012. JDK version is 1.7.0_u25. There is no issue with memory or threads as we analyzed the memory and thread dumps too when the server is non-responsive. Any help would be much appreciated.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
Hello HDK

EJB-JBI bridge always run well in the projects we worked with.  Something wrong must be in your configuration or development. Nevertheless, In order to find an urgent response to your issue, why you don't use SOAP BC to invoke your services. The process will be as follows:
BPEL<-->SOAP BC<--> EJB.

since you are in the same stack he performances must be in the same order that an access through EJB-JBI.

regards

Paul
www.pymma.com The best services on OpenESB
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
Paul,

Thank you for your response.

We tried that but it kills the purpose of using glassfish as our ejb container as the response time increases significantly when we opt for the soap binding to call EJB services.

The reason why we think its an issue related to the java-ee service engine bridge is because all EJBs work fine over 8080 port, which glassfish uses to serve direct responses.


-------- Original message --------
From: "Paul Perez [via OpenESB Community Forum]" <[hidden email]>
Date: 13/01/2016 7:47 pm (GMT+05:00)
To: hkd <[hidden email]>
Subject: Re: Non-Responsive Glassfish/JBI Bridge after some use

Hello HDK

EJB-JBI bridge always run well in the projects we worked with.  Something wrong must be in your configuration or development. Nevertheless, In order to find an urgent response to your issue, why you don't use SOAP BC to invoke your services. The process will be as follows:
BPEL<-->SOAP BC<--> EJB.

since you are in the same stack he performances must be in the same order that an access through EJB-JBI.

regards

Paul
www.pymma.com The best services on OpenESB



To unsubscribe from Non-Responsive Glassfish/JBI Bridge after some use, click here.
NAML
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
I have a simple question for you?
You forecast you will increase dramatically your response time if you pass through a SOAP channel. My question is did you Test it already.
Many of Pymma's customers split EJB and ESB development on Glassfish. They just notice a 1/1000 sec difference between both configuration.
Please test it and let us know

regards

Paul
www.pymma.com The best services on OpenESB
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hammad.karamat
This post has NOT been accepted by the mailing list yet.
Paul,

Yes we tried doing this and please correct me if we did something wrong while doing that:

1. Deploy ejb module separately into glassfish
2. BPEL calls EJBs using the port 8080 and wsdl were opened in browser for each service and saved in a folder in our BPEL module.
3. WSDL is dragged to the right side of bpel process and used.

Response time increases because we have 2 to 4 ejb calls from one BPEL process each on average.


-------- Original message --------
From: "Paul Perez [via OpenESB Community Forum]" <ml-node+[hidden email]>
Date: 13/01/2016 9:47 pm (GMT+05:00)
To: Hammad Karamat Dar <[hidden email]>
Subject: Re: Non-Responsive Glassfish/JBI Bridge after some use

I have a simple question for you?
You forecast you will increase dramatically your response time if you pass through a SOAP channel. My question is did you Test it already.
Many of Pymma's customers split EJB and ESB development on Glassfish. They just notice a 1/1000 sec difference between both configuration.
Please test it and let us know

regards

Paul
www.pymma.com The best services on OpenESB



To start a new topic under OpenESB Community Forum, email ml-node+[hidden email]
To unsubscribe from OpenESB Community Forum, click here.
NAML
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
So that is not normal.

First you have to test the reliability and scalability of your EJB

I propose you to test in charge your EJB  with a tools such as SOAP UI and test your EJB has the same behaviour with something more pushy than your browser.

Let me know

Paul
www.pymma.com The best services on OpenESB
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
Pardon me as i fail to understand what is not normal? If you mean that its not normal to have 3-5 calls for EJBs per BPEL process than i would disagree. We are moving towards a microservices architecture and our process' requirement dictates these number of calls.

EJBs are perfectly fine and run normally when accessed over port 8080, as I mentioned earlier. BPEL threads work fine too as all the calls reach to BPEL without delay and then messages never get picked up from the NMR by Java-ee-se. Please do share your thoughts and suggestion as what should we do. We can not decrease the number of calls rather i am afraid, we might be adding couple of more calls to BPEL processes.



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
I don't find normal when you wrote:
Response time increases because we have 2 to 4 ejb calls from one BPEL process each on average.
My question is: Did you try to invoke your EJB in the same condition but with the SOAP UI.
I would like to be sure that your EJB is scalable

regards

Paul

www.pymma.com The best services on OpenESB
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
This post was updated on .
How can response time not be increased if each EJB call becomes an external call using SOAP binding? I believe that would become an external HTTP call right?

Yes EJBs perform fine as we have performance based logging using the Interceptors which print out the time taken for the EJB to process the request. Let me give you some statistics:

Our LoginProcess, which is a BPEL process, calls 3 EJBs. First to get the user details from one EJB, then based on that, we call another system to get the authentication and authorization related information. Once user is authorized, we log the user in by adding a record in the database too using another EJB.

1. When everything is normal, who process taken 60ms by max.
2. When glassfish is sluggish, EJBs(approx 10,10 and 10ms respectively ) take the same time but the message picked up by the Java-EE-SE takes a minute or more than that. Once picked up, EJB again takes the same time(10, 10 and 10ms).

When we restart the glassfish, it starts working fine again for the next 4-5 days again. When the glassfish is experiencing the issue, we tried calling the EJB WS directly using SOAPUI and all the EJBs responded swiftly and took the same time (10, 10 and 10ms). Even those BPEL processes which have database-bc and are executing direct queries work fine too when the glassfish is sluggish.

I understand that complex processing happens when a call is being forwarded to EJB container(ClassLoaders being changed etc), I think its an issue related to that for some reason. I hope the information would help you to help us out.

Thank you once again for your prompt responses.
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
Any hint guys? I would appreciate if someone points me in the right direction.

Best!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
Hi HDK,

As explain in OpenESB web site, we recommend to submit an issue with a denatured case, this means that you are able to submit a repeatable case.
You mention that your issue occurs after 4-5 days. This means that the problem can come from a memory leak or a connection not close or any kind of things that require a deep and advanced check in.  I hope that someone may be faced the same case and provide you with a straightforward solution. Else, except a psychic, I don't think that someone on the forum can solve your problem without a deeper involvement in time and budget.

I wish you the best

Paul


www.pymma.com The best services on OpenESB
hkd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

hkd
This post has NOT been accepted by the mailing list yet.
Paul,

The issue is not related to any memory leak or connection leak as i mentioned earlier too because all EJBs work fine over 8080 port. All BPEL processes respond correctly, memory and CPU are under-utilized and we have thread dumps and heap dumps to prove that. The issue is right after Message Exchange is created as we can see the log entry of that action. After that, there is nothing in the logs. NMR monitoring tells us that active exchanges are increased but nothing happens.

All the thread-pool-1 threads are waiting:

p: thread-pool-1; w: 46" daemon prio=6 tid=0x000000002c3dd800 nid=0x1c38 in Object.wait() [0x0000000034b9f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000074676a158> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
        at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
        - locked <0x000000074676a158> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

   Locked ownable synchronizers:
        - None

Secondly, I strongly feel that community should be helpful without the support plan subscription too and such no-clue issues would force the users to switch the products.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-Responsive Glassfish/JBI Bridge after some use

Paul Perez
Administrator
Hdk,

I apologise for my previous email since I certainly did not explain my idea in a good way.

First, I never said that your issue is a memory leak but  I tried to explain you that your issue does not seem to be an obvious case and look likes the memory leak bug or thread blocking since they occurs after many days and are not obvious to find and requires deeper investigation.

Second, no one is forced to subscribe to a plan or anything else. Far from me to push you in that way. But regarding your context (Application still in production--> Urgency ), I just give you an advice in order to fix your issue quickly but certainly not to force you to subscribe to any plan subscription. Please feel free to throw my advise to a bin if you want.

I just repeat objective points already mentioned in our conversation before.

Bug Fixing in an open source community is not a due but relies on members focus and availability. During the last years, they fix thousands of bug and help many projects to go forward as much as they can, but unfortunately, there is no guarantee to get a fix immediately when it is needed.  Putting an application in production without support appeared a bit risky especially when the community did not support the version in production any more. My advice was just to help you to reduce the risk your company took.

Again that is just my point of view and nothing else.

I really hope that your issue will be fixed quickly.

Best Regards

Paul

 
www.pymma.com The best services on OpenESB
Loading...