Affecting System - UK Data Centre

• 08/12/2022 08:30
• Last Updated 08/12/2022 19:17

NOTE: The JetBackups sync process is slow, it requires an index of all backups before any restore can occur. If you have your own copies of your website/application and wish to have a clean account created for you please open sales ticket (sales@hostmedia.uk) and the team will create this for you.

UPDATE 9 DISASTER RECOVERY PLAN: We are progressing with the full disaster recovery plan with the following stages which will be updated as we progress:

1. New clean install of OS COMPLETE
2. Installation of the core software:
   - DirectAdmin COMPLETE
     (Note: Implemented new DA version with PRO Pack)
   - CloudLinux COMPLETE
   - LiteSpeed Web Server COMPLETE
   - JetBackups IN PROGRESS
     (SYNCING BACKUP DATA - This is a slow process and being monitored)
3. Link JetBackups with offsite cloud backup servers
4. Begin account restore (this will take some time)
5. During account restore install non-core software
   - Softaculous
   - CrossBox Mail Suite IN PROGRESS
   - Website Builders

Please note the account restores for resellers may require our teams to contact you via a help desk support ticket to ensure any/all sub DirectAdmin accounts are properly synced with your user and are restored correctly.

UPDATE 8: Unfortunately it appears the disk became corrupted due to the constant power disruptions on the Darwin node that we are putting in place our full recovery plans. This will take some time and we will be working with customers to get them back online asap. We will post further updates here once this has been started. We are very sorry for the inconvenience this has certainly caused. A post debrief with the data centre and management will occur where we understand the full details of this.

UPDATE 7: Progress with the DirectAdmin node labelled Darwin is underway and we are attempting to repair the damage caused to the storage drives from the power outage. At the moment we don't have an ETA on the repair but will keep this incident report updated.

UPDATE 6: Most servers are confirmed online, the Darwin node appears to have disk issues which need to be repaired due to the power being suddenly stopped. We are working on this as quickly as possible but disk scans and corrections can take some time.

UPDATE 5: We are servers appearing to still be having issues and servers becoming active and then down again. The data centre are actively working on this and we are monitoring our services closely to get everything restored asap.

UPDATE 4: We are seeing most servers online, but a couple including the Darwin DirectAdmin server remains down. An engineer at the data centre is checking this now.

UPDATE 3: Power has been restored but we are still closely monitoring the situation and awaiting all servers to come back online.

UPDATE 2: Some racks have started to come back online but unfortunately does not include ours. The engineers are investigating the issue and we hope for updates soon. 

UPDATE 1: You can track the data centre updates via https://status.ukservers.com/

We are currently working with our UK data centre on a power issue that is effecting the entire data hall. We will post updates as we get them.