Microsoft Azure (UK South) outage (Opgelost)
  • Prioriteit - Laag
  • Beïnvloeden Overige - Microsoft Azure (UK South) outage
  • Error
    At approx. 14:00 GMT+1 on Monday 14th September 2020 we noticed some of our backup clusters were unavailable.

    Fix
    Microsoft has now resolved the issue and our clusters have returned to operation.

    Cause
    Microsoft has published the following notice:

    Summary of impact: This incident is now mitigated. Between 00:40 UTC on 15 Sep 2020 and 06:06 UTC on 15 Sep 2020, you were identified as a customer using Virtual Machines in UK South who may have experienced connection failures when trying to access some Virtual Machines hosted in the region. These Virtual Machines may have also restarted unexpectedly.

    Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences. 

    Further action
    Microsoft will publish a full incident report on 22/09/2020 and we will post further updates here - https://www.bobcloud.net/blog/microsoft-azure-uk-south-region-outage-on-14th-september-2020/

  • Datum - 14/09/2020 14:00 - 15/09/2020 07:00
  • Laatste update - 17/09/2020 08:40
Wasabi EU region on errror (Opgelost)
  • Prioriteit - Kritisch
  • Beïnvloeden Overige - Storage destination unavailable
  • Error
    Since 1st April 2020 @ 20:00 UTC (21:00 GMT), the Wasabi EU region has been on a degraded error.

    At this time we cannot display the storage buckets in the EU region and we consider it on complete fail. The US regions are unaffected.

    Live updates can be viewed on Wasabi's site - https://status.wasabi.com/

    We are monitoring the situation.

     

    Updated (2nd April @ 14:00)

    On 2nd April 2020 @ approx. 13:00 GMT, we noted BOBcloud backups are working correctly.

    We omitted to mention that this outage also affected our FileSync services. These are running normally now.

    We will continue to monitor this before providing the all-clear.

    Updated (6th April @ 09:00). CLOSED

    All backup sets we were monitoring are now working correctly.

     

  • Datum - 01/04/2020 21:00 - 06/04/2020 09:33
  • Laatste update - 06/04/2020 09:33
Backup portal (Opgelost)
  • Prioriteit - Hoog
  • Beïnvloeden Systeem - https://portal.data-stor.net/
  • The reseller portal can't connect to the backup clusters. Some resellers won't be able to manage their accounts.

    We are looking into this and apologise for the outage.

  • Datum - 24/01/2020 15:06 - 26/01/2020 22:10
  • Laatste update - 26/01/2020 22:09
B23 backup cluster is offline (Opgelost)
  • Prioriteit - Kritisch
  • Beïnvloeden Server - Backup cluster
  • 4th July 2019: 07:23

    We are experiencing an issue with our B23 cluster on Microsoft's cloud.
    This appears to be a Microsoft cloud issue and we are communicating with their support team now. The error is shown below.

    At 07:22 AM, Thursday, 04 July 2019 UTC, your virtual machine b23 became unavailable because it lost connectivity to its remote disk. Azure platform continuously monitors reads and writes (IO transactions) from your virtual machine to Azure Storage service where your virtual machine's disks reside. If transactions do not complete successfully within 120 seconds (inclusive of retries), connectivity is considered to be lost and your virtual machine is shut down temporarily. This is done to preserve data integrity and prevent corruption of your virtual machine. Once Azure platform detects that connectivity to Storage service is restored, your virtual machine is automatically restarted. RDP and SSH connections to your virtual machine, or requests to any services running inside your virtual machine may have failed during this time. We apologize for any inconvenience this may have caused you. We are continuously working on improving the Azure platform to reduce availability incidents.

    We apologise for any inconvenience.

    ------------

    09:02
    Microsoft Support has confirmed this is a regional issue and other customers have reported the outage.

    ------------

    09:12
    The service has been restored and we continue to monitor this.

    ------------

    09:35
    The service has failed again.

    To prevent any false reporting, we will only clear the error 30 minutes after it has been marked as resolved.

    ------------

    12:35
    Azure have  updated their status page https://status.azure.com/en-us/status

    ------------

    14:36
    This issue has been resolved. Please open a new ticket if you are experiencing issues.

    BOBcloud NOC

     

     

     

  • Datum - 04/07/2019 08:29
  • Laatste update - 04/07/2019 14:37
Logon problems - https://portal.data-stor.net (Opgelost)
  • Prioriteit - Gemiddeld
  • Beïnvloeden Overige - BOBcloud logon
  • We are experiencing an issue with logons for the backup portal at https://portal.data-stor.net

    Please use this URL instead until further notice: https://uk.servermanagerpro.com

    We apologise for any inconvenience.

    BOBcloud NOC

  • Datum - 28/01/2019 07:43 - 24/03/2019 19:27
  • Laatste update - 28/01/2019 07:45
Moved to forum (Opgelost)
  • Prioriteit - Gemiddeld
  • Beïnvloeden Overige - Ahsay version 7
  • Moved to https://forum.bobcloud.net/t/known-issues-awaiting-bug-fixing/479

    Issue

    Over quota usage on version 7 doesn't always prevent backups from running.
    Version 7 accounts are on clusters b21.data-stor.net and upwards.
    You can check which cluster an account is using from the right hand side of their customer page in your reseller portal.

    quota-error


    Impact
    When a quota is reached, the system might not prevent any further backups and consequently you could overuse the service.

    Fix
    We have logged this issue with Ahsay and it is likely to be fixed in early 2018.

     

  • Datum - 17/10/2017 09:48
  • Laatste update - 03/03/2018 13:57
Server repair on cluster B7 (Opgelost)
  • Prioriteit - Kritisch
  • Beïnvloeden Overige - B7
  • We have taken offline one of the servers in our B7 cluster to repair profiles of some affected customers.
    This is a result of data corruption found after November's power outage at our Reading facility.
    We expect the server to be online some time on the morning of Monday 12th and in time for overnight backups.

    11:59
    The repair is taking longer than expected and is still running.

    20:01
    The repair has now finished and the cluster is back online.

  • Datum - 11/12/2016 22:23 - 12/12/2016 11:59
  • Laatste update - 13/12/2016 00:01
Cluster server repair of B10 (Opgelost)
  • Prioriteit - Gemiddeld
  • Beïnvloeden Overige - B10
  • Following on from Tuesday's power outage, we have taken one of our cluster servers offline to repair profiles of some affected customers.
    We will post updates on the progress.

    14:00
    This issue has now been resolved.

  • Datum - 19/11/2016 10:29 - 19/11/2016 14:00
  • Laatste update - 11/12/2016 14:24
Backup cluster B3 (Opgelost)
  • Prioriteit - Kritisch
  • Beïnvloeden Overige - B3
  • We are currently experiencing connectivity issues with one of our backup clusters named B3
    This is being looked into.

  • Datum - 03/02/2016 09:50 - 03/02/2016 20:43
  • Laatste update - 19/11/2016 10:34
Everest data centre in Reading outage (Opgelost)
  • Prioriteit - Kritisch
  • Beïnvloeden Overige - All UK services (except AWS and Azure)
  • Our Data Centre provider [Everest http://everestdc.com/] is experiencing a major outage in Reading. We will update the status shortly.

    08:51
    Everest have suffered a major fault with the power system. They have engineers working on the problem, and will provide further details as soon as possible.

    09:11
    Update from Everest.

    There has been an HV mains failure to the site, on both the feeds. We believe this is a local supply issue from the grid. Our backup generators started automatically as per planned operating procedure, but went into fault. We have started the generators manually which are currently powering up.

    10:32
    The power has been restored and our services are coming online. We will post further updates shortly.

    11:39
    Everest have reported the network is back to normal.
    VPS' and some of our services are running again.
    We are recovering from issues causes by the power outage and will post further updates.

    12:30
    We are experiencing issues with one of our upstream routers. We have contacted Cisco and they are sending an Engineer to site. ETA 2 hours

    14:57
    Engineer on site

    16:53
    The network issue has been resolved.
    There are some Ahsay backup issues arising from a sudden power failure earlier and we are looking at these now.

    20:19
    All services are now back to normal.
    We anticipate some Ahsay backups failing or taking longer to run overnight. This is because some of the index files were corrupted when the power went off.
    The Ahsay software has built-in rebuild tools which will fix these problems automatically.
    If you experience any further issues, please open a ticket in the normal way.

    We will deliver a full report in the next few days and apologise for the outage.



  • Datum - 15/11/2016 08:44
  • Laatste update - 19/11/2016 10:33