Quantcast

Performance Issues Looping and Assigning

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Performance Issues Looping and Assigning

PaulCB
Hi,


I have a BPEL flow running in OpenESB standalone (Logicoy version). The flow loops through 1000 XML child elements assigning attributes with each one to another document:

<forEach name="ForEachRecord" parallel="no" counterName="counter">            
            <startCounterValue>1</startCounterValue>
            <finalCounterValue>count($BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords)</finalCounterValue>
            <scope name="Scope1">
                <assign name="AssignRecord">
                    <sxt:trace>
                        <sxt:log level="finest" location="onStart">
                            <from>'Row'</from>
                        </sxt:log>
                    </sxt:trace>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:TransactionRecordId</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:TransactionRecordId</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Source</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Source</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Destination</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Destination</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:StartDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:StartDate</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:EndDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:EndDate</to>
                    </copy>                    
                </assign>
            </scope>
        </forEach>


For 1000 elements it takes about 18s to do this loop on a top end server. My logging shows that at first each assign takes less than 1ms and then slowly goes up. By the time it gets to loop 1000 it's taking over 30ms for an assign.

So it seems like the assign is very inefficient for larger XML documents - maybe its re-parsing the entire document on every assign as it looks for each row to assign?

Any ideas what can be done to optimise this? My use case is for an account history query which can return many rows in the XML and needs to assign the result row by row.

Thanks 

--
Paul Carter-Brown
Group Chief Information Officer
Smile Communications Pty (Ltd) 
      

Smile +234 (0) 702 000 1234
Mobile +27 (0) 83 4427 179
Skype PaulC-B
[hidden email]
www.smilecoms.com

This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance Issues Looping and Assigning

hammad.karamat
This post has NOT been accepted by the mailing list yet.
Hello Paul,

BPEL keeps the state of each iteration in memory for multiple purposes like compensation etc. The approach is not correct and you should avoid loops in BPEL as much as you can. However, there are multiple ways to work around this in OpenESB:

1. Use database-bc to fetch the records and return those directly to the caller.

2. Use stateless EJBs as WS if you need to perform actions on iterations being returned from database or any other source. Call that EJB from BPEL.

3. Use POJO-se to do the same if you do not need transactions.

We tried all three options and achieved considerable performance gains in our application as per our need.

Regards,

Hammad Dar


-------- Original message --------
From: "PaulCB [via OpenESB Community Forum]" <ml-node+[hidden email]>
Date: 17/06/2016 10:56 pm (GMT+05:00)
To: Hammad Karamat Dar <[hidden email]>
Subject: Performance Issues Looping and Assigning

Hi,


I have a BPEL flow running in OpenESB standalone (Logicoy version). The flow loops through 1000 XML child elements assigning attributes with each one to another document:

<forEach name="ForEachRecord" parallel="no" counterName="counter">            
            <startCounterValue>1</startCounterValue>
            <finalCounterValue>count($BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords)</finalCounterValue>
            <scope name="Scope1">
                <assign name="AssignRecord">
                    <sxt:trace>
                        <sxt:log level="finest" location="onStart">
                            <from>'Row'</from>
                        </sxt:log>
                    </sxt:trace>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:TransactionRecordId</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:TransactionRecordId</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Source</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Source</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Destination</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Destination</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:StartDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:StartDate</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:EndDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:EndDate</to>
                    </copy>                    
                </assign>
            </scope>
        </forEach>


For 1000 elements it takes about 18s to do this loop on a top end server. My logging shows that at first each assign takes less than 1ms and then slowly goes up. By the time it gets to loop 1000 it's taking over 30ms for an assign.

So it seems like the assign is very inefficient for larger XML documents - maybe its re-parsing the entire document on every assign as it looks for each row to assign?

Any ideas what can be done to optimise this? My use case is for an account history query which can return many rows in the XML and needs to assign the result row by row.

Thanks 

--
Paul Carter-Brown
Group Chief Information Officer
Smile Communications Pty (Ltd) 
      

Smile +234 (0) 702 000 1234
Mobile +27 (0) 83 4427 179
Skype PaulC-B
[hidden email]
www.smilecoms.com

This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/



If you reply to this email, your message will be added to the discussion below:
http://openesb-community-forum.794670.n2.nabble.com/Performance-Issues-Looping-and-Assigning-tp7581313.html
To start a new topic under OpenESB Community Forum, email ml-node+[hidden email]
To unsubscribe from OpenESB Community Forum, click here.
NAML
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance Issues Looping and Assigning

PaulCB
In reply to this post by PaulCB
Hi,

Due to the fact that I was copying between two complex types that had the same structure but different namespaces, I implemented a hack that marshals the node to a String, passes it to a Java function that changes the namespace and then unmarshal back to the target node.

Went from 18s down to less than 100ms:

BPEL snippet:

<assign name="BMToSCA">
            <copy>
                <from>BPELUtils:changeNamespace(sxxf:doMarshal($BMGetAccountHistoryOut.AccountHistory), 'http://xml.smilecoms.com/schema/BM', 'http://xml.smilecoms.com/schema/SCA')</from>
                <to>$TmpString</to>
            </copy>
            <copy>
                <from>sxxf:doUnMarshal($TmpString)</from>
                <to variable="SCAGetAccountHistoryOut" part="AccountHistory"/>
            </copy>
</assign>

Java snippet:

public static String changeNamespace(String src, String fromNameSpace, String toNameSpace) throws Exception {
        return src.replaceFirst(fromNameSpace, toNameSpace);
}

Not very elegant but you can't argue with the results!

Paul

On 17 June 2016 at 18:37, Paul Carter-Brown <[hidden email]> wrote:
Hi,


I have a BPEL flow running in OpenESB standalone (Logicoy version). The flow loops through 1000 XML child elements assigning attributes with each one to another document:

<forEach name="ForEachRecord" parallel="no" counterName="counter">            
            <startCounterValue>1</startCounterValue>
            <finalCounterValue>count($BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords)</finalCounterValue>
            <scope name="Scope1">
                <assign name="AssignRecord">
                    <sxt:trace>
                        <sxt:log level="finest" location="onStart">
                            <from>'Row'</from>
                        </sxt:log>
                    </sxt:trace>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:TransactionRecordId</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:TransactionRecordId</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Source</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Source</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:Destination</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:Destination</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:StartDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:StartDate</to>
                    </copy>
                    <copy>
                        <from>$BMGetAccountHistoryOut.AccountHistory/ns1:TransactionRecords[$counter]/ns1:EndDate</from>
                        <to>$SCAGetAccountHistoryOut.AccountHistory/ns0:TransactionRecords[$counter]/ns0:EndDate</to>
                    </copy>                    
                </assign>
            </scope>
        </forEach>


For 1000 elements it takes about 18s to do this loop on a top end server. My logging shows that at first each assign takes less than 1ms and then slowly goes up. By the time it gets to loop 1000 it's taking over 30ms for an assign.

So it seems like the assign is very inefficient for larger XML documents - maybe its re-parsing the entire document on every assign as it looks for each row to assign?

Any ideas what can be done to optimise this? My use case is for an account history query which can return many rows in the XML and needs to assign the result row by row.

Thanks 

--
Paul Carter-Brown
Group Chief Information Officer
Smile Communications Pty (Ltd) 
      

Smile <a href="tel:%2B234%20%280%29%20702%20000%201234" value="+2347020001234" target="_blank">+234 (0) 702 000 1234
Mobile +27 (0) 83 4427 179
Skype PaulC-B
[hidden email]
www.smilecoms.com



--
Paul Carter-Brown
Group Chief Information Officer
Smile Communications Pty (Ltd) 
      

Smile +234 (0) 702 000 1234
Mobile +27 (0) 83 4427 179
Skype PaulC-B
[hidden email]
www.smilecoms.com

This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
Loading...