Network Status News & Information

Error
At approx. 14:00 GMT+1 on Monday 14th September 2020 we noticed some of our backup clusters were unavailable.

Fix
Microsoft has now resolved the issue and our clusters have returned to operation.

Cause
Microsoft has published the following notice:

Summary of impact: This incident is now mitigated. Between 00:40 UTC on 15 Sep 2020 and 06:06 UTC on 15 Sep 2020, you were identified as a customer using Virtual Machines in UK South who may have experienced connection failures when trying to access some Virtual Machines hosted in the region. These Virtual Machines may have also restarted unexpectedly.

Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences. 

Further action
Microsoft will publish a full incident report on 22/09/2020 and we will post further updates here - https://www.bobcloud.net/blog/microsoft-azure-uk-south-region-outage-on-14th-september-2020/

Error
Since 1st April 2020 @ 20:00 UTC (21:00 GMT), the Wasabi EU region has been on a degraded error.

At this time we cannot display the storage buckets in the EU region and we consider it on complete fail. The US regions are unaffected.

Live updates can be viewed on Wasabi's site - https://status.wasabi.com/

We are monitoring the situation.

Updated (2nd April @ 14:00)

On 2nd April 2020 @ approx. 13:00 GMT, we noted BOBcloud backups are working correctly.

We omitted to mention that this outage also affected our FileSync services. These are running normally now.

We will continue to monitor this before providing the all-clear.

Updated (6th April @ 09:00). CLOSED

All backup sets we were monitoring are now working correctly.

The reseller portal can't connect to the backup clusters. Some resellers won't be able to manage their accounts.

We are looking into this and apologise for the outage.

4th July 2019: 07:23

We are experiencing an issue with our B23 cluster on Microsoft's cloud.
This appears to be a Microsoft cloud issue and we are communicating with their support team now. The error is shown below.

At 07:22 AM, Thursday, 04 July 2019 UTC, your virtual machine b23 became unavailable because it lost connectivity to its remote disk. Azure platform continuously monitors reads and writes (IO transactions) from your virtual machine to Azure Storage service where your virtual machine's disks reside. If transactions do not complete successfully within 120 seconds (inclusive of retries), connectivity is considered to be lost and your virtual machine is shut down temporarily. This is done to preserve data integrity and prevent corruption of your virtual machine. Once Azure platform detects that connectivity to Storage service is restored, your virtual machine is automatically restarted. RDP and SSH connections to your virtual machine, or requests to any services running inside your virtual machine may have failed during this time. We apologize for any inconvenience this may have caused you. We are continuously working on improving the Azure platform to reduce availability incidents.

We apologise for any inconvenience.

------------

09:02
Microsoft Support has confirmed this is a regional issue and other customers have reported the outage.

------------

09:12
The service has been restored and we continue to monitor this.

------------

09:35
The service has failed again.

To prevent any false reporting, we will only clear the error 30 minutes after it has been marked as resolved.

------------

12:35
Azure have  updated their status page https://status.azure.com/en-us/status

------------

14:36
This issue has been resolved. Please open a new ticket if you are experiencing issues.

BOBcloud NOC

We are experiencing an issue with logons for the backup portal at https://portal.data-stor.net

Please use this URL instead until further notice: https://uk.servermanagerpro.com

We apologise for any inconvenience.

BOBcloud NOC

Moved to https://forum.bobcloud.net/t/known-issues-awaiting-bug-fixing/479

Issue
Over quota usage on version 7 doesn't always prevent backups from running.
Version 7 accounts are on clusters b21.data-stor.net and upwards.
You can check which cluster an account is using from the right hand side of their customer page in your reseller portal.




Impact
When a quota is reached, the system might not prevent any further backups and consequently you could overuse the service.

Fix
We have logged this issue with Ahsay and it is likely to be fixed in early 2018.

We have taken offline one of the servers in our B7 cluster to repair profiles of some affected customers.
This is a result of data corruption found after November's power outage at our Reading facility.
We expect the server to be online some time on the morning of Monday 12th and in time for overnight backups.

11:59
The repair is taking longer than expected and is still running.

20:01
The repair has now finished and the cluster is back online.

Following on from Tuesday's power outage, we have taken one of our cluster servers offline to repair profiles of some affected customers.
We will post updates on the progress.

14:00
This issue has now been resolved.

We are currently experiencing connectivity issues with one of our backup clusters named B3
This is being looked into.

Our Data Centre provider [Everest http://everestdc.com/] is experiencing a major outage in Reading. We will update the status shortly.

08:51
Everest have suffered a major fault with the power system. They have engineers working on the problem, and will provide further details as soon as possible.

09:11
Update from Everest.

There has been an HV mains failure to the site, on both the feeds. We believe this is a local supply issue from the grid. Our backup generators started automatically as per planned operating procedure, but went into fault. We have started the generators manually which are currently powering up.

10:32
The power has been restored and our services are coming online. We will post further updates shortly.

11:39
Everest have reported the network is back to normal.
VPS' and some of our services are running again.
We are recovering from issues causes by the power outage and will post further updates.

12:30
We are experiencing issues with one of our upstream routers. We have contacted Cisco and they are sending an Engineer to site. ETA 2 hours

14:57
Engineer on site

16:53
The network issue has been resolved.
There are some Ahsay backup issues arising from a sudden power failure earlier and we are looking at these now.

20:19
All services are now back to normal.
We anticipate some Ahsay backups failing or taking longer to run overnight. This is because some of the index files were corrupted when the power went off.
The Ahsay software has built-in rebuild tools which will fix these problems automatically.
If you experience any further issues, please open a ticket in the normal way.

We will deliver a full report in the next few days and apologise for the outage.