1^st.

We apologize for the inconvenience and will ensure to keep you updated throughout the incident via mails.

2^nd.

To System1 universe users:

Please be advised that an issue has occurred with the connection between System1 and the source database in System2. This is under investigation currently and is being treated with the highest priority. At this time we do not have and estimated time for resolution.

In relation to this incident below pls find latest update on the situation:

3^rd.

Please note that the issue was discovered to be with the account on the System1 database. We believe now that this has been resolved and you should now be able run your System1 reports in System2. Thank you for your patience during this issue.

Regards,

4^th.

Be advised that we are well aware of the recurrence of this issue. The below message from the Prime Mover of System1 describes why this issue has reoccurred. With this stated, whomever has increased reporting in System2 in the past few day, PLEASE CEASE THIS NEW REPORTING IMMEDIATELY as System2 connection to System1 has been deliberately blocked based on the sudden increase in reporting activity occurring in the System1 database.

“We have severe issues on System1 today and it seems it is caused by users pulling a large amount of reports using BO. We have seen an increase of 200% and it is pulling down System1 performance. In order to restore performance IBM has blocked BO so your users will not be able to pull anything on System1. The same happened yesterday and I am very sorry you were not notified of this. I should have told you and IBM should have told IBM US. I am very sorry for the inconvenience caused to you and your team on this. The connection will be resorted once System1 is stable again. Yesterday it was 18 CET – I do not hope it will take this long today.

Right now we are investigating why users are pulling more reports than usual and I have been in contact with the user’s but we have not reached any conclusion. Do you know of any abnormal behavior or have heard of any projects going on at the moment that could cause this?”

If anyone can provide any information as why the sudden increase in reporting in System2 please kindly advise by reply to the DL in Cc. Otherwise, please kindly do not respond to this message.

Thank you,

5^th.

System2’s access to BO has now been restored. We believe we have identified a possible root cause for the increased load, and have asked this team to suspend this process tomorrow to see if it truly resolves the issue.  Thank you again for your patience during this.

Regards

6^th.

We believe we have found the user source of the spike in reporting and have disabled this user for the time being. Be advise that if there is further over heightened reporting that System1 will be forced to lock out the System2.

Regards

7^th.

To System1 Export universe users in System2:

Please note that System2 was back and was working normal, however again user's are getting the same error message.

We have informed the same thing to the vendor, however till now haven't received any expected timeline.

Already have raised P1 Incident - for this case.

Really apologies for the inconvenience caused and will ensure to keep you updated throughout the mails.

Note : Please do not raise separate Remedy ticket for this issue

Best regards

8^th.

Hello.

Be advise that due to the session increase significantly to 103 and the CPU idle% dropped to 15%, USER ID was locked as of 8:40 CET.  

Now unlocked the User ID as of 11:40 CET and are monitoring the system for performance.

Thank you again for your patience.

Note : Please do not raise separate Remedy ticket for this issue

9^th.

All System1 Universe Users in BO,

Based on prior correspondence on this issue, you may received the following message if you have tried to run a System1 report this week:

Database error: ORA-xxxxx

I would like to clarify that, while you are receiving this message, it is not an error but is being done by design by the System1 application team to protect the System1 application itself. 

The reason is there has been a 30% increase in System1 reporting this week which is having a negative impact on the System1 application. 

We are working to identify which processes have changed this week and/or what new reports are being run that contribute to this issue; however, merely blocking some users and stopping some new processes has not had enough of a positive impact.  Therefore a joint decision was reached with the System1 application team that they will monitor the load which is being used by the System1 universes; if it exceeds a critical level they will lock the connection from the universes to System1 until the peak hours have passed.  This can affect multiple users and will continue until sufficient load is removed or better distributed throughout the day.

While we will be using usage statistics to try and track down some of the heavier users and most refreshed reports we also need your help in order to reduce the load.

·         If you are running a report for someone else please confirm they still need it.

·         If you are running a report and would like to move it to a System2 universe but not sure if possible please reach out to

·          

·         If you are running reports using a macro (i.e. Auto BO) please reach out to me directly, and assistance will be given to identify a more efficient process.

·         If you are using the System1 universe to source a database of some kind, please reach out to me directly; assistance will be given in moving toward a more sustainable solution.

·         If you are using System1 reports for Cargo Readiness (previously Vessel Reconciliation) please switch to the new global solution for this process which is available in the CODS Web Portal. (Note: Simply login with your CRB details.)

·         If you aware of anything not mentioned above which could be causing increased load please also let me know.

We understand the inconvenience this is causing, but you should anticipate continued outages between 0500 UTC and 0900 UTC daily until we are able to reduce the usage of the System1 universes. 

NOTE: We will be closing all related Remedy incidents.  Please do not open any new incidents or escalate the closing of the current ones.

If you have any questions, comments or concerns on the above, I encourage you to reply to me directly.  I again apologize for any inconvenience this may cause however this is in the best interest of the organization.  I look forward to a collaborative effort throughout the world to reduce the load to a more acceptable effort.

Regards,