 |
Olympus ODDS May-June 2020 Outage What Happened?
From May 28th to June 10th the Olympus ODDS service either delayed sending of audio for hours or days or was completely down. The server which receives and then distributes all the dictation from the Olympus Dictation App globally reached its limits and was unable to scale mainly due to increased use from COVID-19.
Olympus App Outage Lessons Learnt?
- Having one server processing the worlds dictation is a single point of failure and needs to either scale or split to regional systems
- Dictation was not able to be taken off the app frustrating users, if the audio could have been AirDrop’ed or emailed from the app would have eased the situation
- Communication. Apologies Olympus Japan but you needed to communicate with your ODDS users. The users of the app came to their dealers, often overwhelming us with phone calls and emails. We went to our regional Olympus reps who then waited on news from Japan. A tweet, updates on a website, even emails to ODDS Authors (the email addresses are all in the ODDS system) would have helped.
Olympus Resolution to the ODDS Issues
Olympus stated in one of their letters that a plan would be communicated on June 12th. It didn’t come until June 24th, here it is:
===================================================
Olympus would like to further express its apologies for the recent ODDS server outages by providing a 2 month license extension for all users who are subscribed to the system as at 1st July 2020.
All users with active licenses as of 1st July will have their expiry date extended by 2 months.
For users with licenses that require renewal from 1st July to the 14th July, the extension will be provided on 1st August, once they complete the subscription for a further year.
To ensure that the service remains operating under ever increasing traffic or any other unexpected situation, Olympus have committed to put in place the following measures:
1. Increase capacity on the ODDS to ensure the system can cope with ever increasing demand.
2. Make Improvements to the backup/redundancy processes to maintain system continuity.
3. Introduce increased monitoring processes to the Server.
4. Implement a function to notify end users through the App and/or Email notification in case the Service is down or compromised in any way.
Olympus thanks you for your patience and appreciates the continued support of its customers.
===================================================
So better monitoring, better communication via direct email and two-month extension to your ODDS subscription licences.
The Olympus App Outage Details:
June 2020 is a month that most users of the Olympus Dictation App with an ODDS (Olympus Dictation Delivery Service) will want to forget as the “Delivery Service” part of the deal failed. Over the course of two weeks the service was either in a state of major dictation delivery delay or down completely, at one point for four days for maintenance.
The issues started at the end of May when we started to get reports of audio “Waiting to Send” in the app. A number of our users reported that they had been experiencing issues for weeks in the build-up to the outage so something was clearly wrong behind the scenes. By June 4th Olympus released a letter to the dealers to pass on to their customers, this was a global outage. Dates in the letter refer to European time so Australia was one day ahead.
===================================================
Dear Olympus Dictation Customers,
Subject: Olympus Dictation App
As we all adjust to this new normal, I hope you are staying safe and healthy. I understand that you have experienced issues with Olympus’ dictation service that have been occurring of late and I would like to deeply apologise to you for any inconvenience caused by this on behalf of the Olympus Pro Audio global team.
Olympus regrets to inform you that all services for the Olympus Dictation App and Olympus Dictation Portal (ODP) are now suspended from today until Sunday, June 6th for maintenance purposes.
A maintenance process had been conducted in the background this week to clear unnecessary data accumulated on the server. This however caused a large number of error messages to be sent out to users, resulting in the server becoming unstable. According to the latest R&D investigation, there may have been a chance that dictations could become lost. Therefore we have decided to suspend the service in order to protect the dictations.
The root cause of the issue has been identified and we are implementing measures to prevent further problems. Maintenance will be conducted in parallel to recover the stability of the service and we expect this to be completed by Sunday at the latest.
For any dictations recently sent, we advise users to keep the dictation on their devices until the service resumes, to ensure that the user’s dictations are not lost. Once the service resumes, dictations can be resent to the recipients.
Olympus will provide a further update when the system resumes on Sunday.
We appreciate the patience shown by all our customers and once again apologise for any inconvenience caused by this service outage.
Sincerely yours,
Ichiro Igari
Director
Global Sales and Marketing Planning
===================================================
Luckily for most in Australia the four-day maintenance window, due to end Monday evening Australia time, was over a weekend which included a public holiday. Olympus, in the end, did not need the full four days and on Sunday June 7th at 11am we tweeted an email from Olympus:
Here is the full letter from Olympus released to global dealers on 7th June 2020:
===================================================
Dear Olympus Dictation Customers,
Subject: Operation recovered for the Olympus Dictation App
June 7th, 2020
We would like to inform you that all services for the Olympus Dictation App and Olympus Dictation Portal has now recovered, as of June 7th, 10:00 JST.
Due to the recent increase of dictations being processed on the Olympus Dictation Delivery Service, the interaction within the servers has become unstable, causing unexpected errors and delaying the delivery of dictations. During the past few days, we have implemented a number of new functions to clear the bottleneck and to prevent the system to overflow. At the same time, all maintenance has been completed to clear the unnecessary data accumulated by the error and the server has recovered the stability.
Dictations kept on your device may now be sent to your recipients. In case you are not sure if your previous dictations reached your destination, we recommend that you send your dictation again.
In order to provide reliable and sustainable service for our customers, we are considering to take further measures to ensure that the service remains running under any circumstances. We will update you on our plan on June 12 .
Once again, we apologise for any inconvenience caused by the service outage.
Sincerely yours,
Ichiro Igari
Director
Global Sales and Marketing Planning Imaging Business Unit
Olympus Corporation
===================================================
Olympus App users and dealers globally felt some relief and looked forward to things going back to normal. That wasn’t the case and very quickly the system ground to a halt again, the result of a few days of global dictation backlog hitting the ODDS system. An update was promised June 12 with a plan moving forward, this never came as fire fighting was under way again. Two days after the maintenance shutdown the system was again stopped as reported by Olympus’ European Support Team:
By June 10th frustration amongst our customers had reached boiling point. We tried our best to keep people updated, by the time we emailed to notify things were back dictations were being delayed again. The biggest issue with customers was that their audio was stuck on the app, no way to bypass the server settings and either email or in the case of iPhone users airdrop their audio. Of course many had no contingency, why would they the system worked well since 2013. I think one email from one of our customers summed up the situation well:
All up the system was unstable or unusable for just under two weeks. For most users, legal and medical, the Olympus Dictation App has shown itself to be an essential tool of the trade. Going forward we are sure that this event was something that needed to happen to re-focus Olympus on the excellent ODDS system. It is an essential tool for so many, we would like to see the ODDS system split up to remove the single point of failure with one server and have dedicated regional instances. Some of our Federal and Government clients have been requesting this for some time and makes sense for mainly confidential audio to stay in-country.