Generator #3 Start Failure : (Equinix - AM11) (Resolved)
  • Priority - Low
  • Malfunction Other - DC-01
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about following incident by one of our Datacenter providers.

    INCIDENT SUMMARY: Generator 3 Start Failure

    INCIDENT DESCRIPTION:

    Equinix IBX Site Staff reports that during biweekly generator no load testing a starting failure was discovered on Generator 3. The vendor has been contacted and is en route. IBX Engineers are investigating the issue.

    Situation Status:  Stable

    Equipment/Condition:  Generator 3 Start Failure

    Current System Redundancy:  Reduced Redundancy

    Resources on Site/Pending:  Vendor en Route

    Generator Fuel Run Time (hrs.):  N/A

    Current Colo Temperature Status: N/A

    Current Colo Humidity RH Status:  N/A

    UPDATE at 12:49 PM :

    Equinix IBX Site Staff reports that upon investigation, the vendor found a loose control wire. The wire was reconnected and Generator 3 was tested successfully. Other connections were also tested and the redundancy is back to normal at N+1.

    This issue is now resolved and service returned to normal as of 29-APR-2020 12:28 Site Local Time.

    If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

     

    Best regards,

    Team 3W INFRA

     

  • Date - 29/04/2020 11:09 - 29/04/2020 12:49
  • Last Updated - 29/04/2020 18:17
Scheduled Expansion of Generator system : (Equinix - AM11) (Resolved)
  • Priority - Low
  • Malfunction Other - DC-01
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about the upcoming maintenance by one of our Datacenter providers.

    DESCRIPTION: Please be advised that Equinix will be reconfiguring the generator backup installations and add an additional generator to increase the site capacity. The project involves several load transfers between main distribution A and B and will be completed with full testing and commissioning of the new setup and an black out test where a controlled shutdown of the incoming power supply to the facility will be performed, simulating a failure of the utility supply and confirming all emergency power systems operates correctly. 

    During these works customers loads will remain on UPS supplied power at all times. The generator redundancy is during the works reduced to “N”

    DATE: 04-MAR-2020 - 10-APR-2020

    SPAN: 04-MAR-2020 - 10-MAR-2020

    LOCAL: WEDNESDAY, 04 MAR 07:00 - WEDNESDAY, 04 MAR 17:00
    UTC: WEDNESDAY, 04 MAR 06:00 - WEDNESDAY, 04 MAR 16:00

    LOCAL: THURSDAY, 05 MAR 07:00 - THURSDAY, 05 MAR 17:00
    UTC: THURSDAY, 05 MAR 06:00 - THURSDAY, 05 MAR 16:00

    LOCAL: FRIDAY, 06 MAR 07:00 - FRIDAY, 06 MAR 17:00
    UTC: FRIDAY, 06 MAR 06:00 - FRIDAY, 06 MAR 16:00

    LOCAL: SATURDAY, 07 MAR 07:00 - SATURDAY, 07 MAR 17:00
    UTC: SATURDAY, 07 MAR 06:00 - SATURDAY, 07 MAR 16:00

    LOCAL: SUNDAY, 08 MAR 07:00 - SUNDAY, 08 MAR 17:00
    UTC: SUNDAY, 08 MAR 06:00 - SUNDAY, 08 MAR 16:00

    LOCAL: MONDAY, 09 MAR 07:00 - MONDAY, 09 MAR 17:00
    UTC: MONDAY, 09 MAR 06:00 - MONDAY, 09 MAR 16:00

    LOCAL: TUESDAY, 10 MAR 07:00 - TUESDAY, 10 MAR 17:00
    UTC: TUESDAY, 10 MAR 06:00 - TUESDAY, 10 MAR 16:00

    SPAN: 11-MAR-2020 - 17-MAR-2020

    LOCAL: WEDNESDAY, 11 MAR 07:00 - WEDNESDAY, 11 MAR 17:00
    UTC: WEDNESDAY, 11 MAR 06:00 - WEDNESDAY, 11 MAR 16:00

    LOCAL: THURSDAY, 12 MAR 07:00 - THURSDAY, 12 MAR 17:00
    UTC: THURSDAY, 12 MAR 06:00 - THURSDAY, 12 MAR 16:00

    LOCAL: FRIDAY, 13 MAR 07:00 - FRIDAY, 13 MAR 17:00
    UTC: FRIDAY, 13 MAR 06:00 - FRIDAY, 13 MAR 16:00

    LOCAL: SATURDAY, 14 MAR 07:00 - SATURDAY, 14 MAR 17:00
    UTC: SATURDAY, 14 MAR 06:00 - SATURDAY, 14 MAR 16:00

    LOCAL: SUNDAY, 15 MAR 07:00 - SUNDAY, 15 MAR 17:00
    UTC: SUNDAY, 15 MAR 06:00 - SUNDAY, 15 MAR 16:00

    LOCAL: MONDAY, 16 MAR 07:00 - MONDAY, 16 MAR 17:00
    UTC: MONDAY, 16 MAR 06:00 - MONDAY, 16 MAR 16:00

    LOCAL: TUESDAY, 17 MAR 07:00 - TUESDAY, 17 MAR 17:00
    UTC: TUESDAY, 17 MAR 06:00 - TUESDAY, 17 MAR 16:00

    SPAN: 18-MAR-2020 - 24-MAR-2020

    LOCAL: WEDNESDAY, 18 MAR 07:00 - WEDNESDAY, 18 MAR 17:00
    UTC: WEDNESDAY, 18 MAR 06:00 - WEDNESDAY, 18 MAR 16:00

    LOCAL: THURSDAY, 19 MAR 07:00 - THURSDAY, 19 MAR 17:00
    UTC: THURSDAY, 19 MAR 06:00 - THURSDAY, 19 MAR 16:00

    LOCAL: FRIDAY, 20 MAR 07:00 - FRIDAY, 20 MAR 17:00
    UTC: FRIDAY, 20 MAR 06:00 - FRIDAY, 20 MAR 16:00

    LOCAL: SATURDAY, 21 MAR 07:00 - SATURDAY, 21 MAR 17:00
    UTC: SATURDAY, 21 MAR 06:00 - SATURDAY, 21 MAR 16:00

    LOCAL: SUNDAY, 22 MAR 07:00 - SUNDAY, 22 MAR 17:00
    UTC: SUNDAY, 22 MAR 06:00 - SUNDAY, 22 MAR 16:00

    LOCAL: MONDAY, 23 MAR 07:00 - MONDAY, 23 MAR 17:00
    UTC: MONDAY, 23 MAR 06:00 - MONDAY, 23 MAR 16:00

    LOCAL: TUESDAY, 24 MAR 07:00 - TUESDAY, 24 MAR 17:00
    UTC: TUESDAY, 24 MAR 06:00 - TUESDAY, 24 MAR 16:00

    SPAN: 25-MAR-2020 - 31-MAR-2020

    LOCAL: WEDNESDAY, 25 MAR 07:00 - WEDNESDAY, 25 MAR 17:00
    UTC: WEDNESDAY, 25 MAR 06:00 - WEDNESDAY, 25 MAR 16:00

    LOCAL: THURSDAY, 26 MAR 07:00 - THURSDAY, 26 MAR 17:00
    UTC: THURSDAY, 26 MAR 06:00 - THURSDAY, 26 MAR 16:00

    LOCAL: FRIDAY, 27 MAR 07:00 - FRIDAY, 27 MAR 17:00
    UTC: FRIDAY, 27 MAR 06:00 - FRIDAY, 27 MAR 16:00

    LOCAL: SATURDAY, 28 MAR 07:00 - SATURDAY, 28 MAR 17:00
    UTC: SATURDAY, 28 MAR 06:00 - SATURDAY, 28 MAR 16:00

    LOCAL: SUNDAY, 29 MAR 07:00 - SUNDAY, 29 MAR 17:00
    UTC: SUNDAY, 29 MAR 05:00 - SUNDAY, 29 MAR 15:00

    LOCAL: MONDAY, 30 MAR 07:00 - MONDAY, 30 MAR 17:00
    UTC: MONDAY, 30 MAR 05:00 - MONDAY, 30 MAR 15:00

    LOCAL: TUESDAY, 31 MAR 07:00 - TUESDAY, 31 MAR 17:00
    UTC: TUESDAY, 31 MAR 05:00 - TUESDAY, 31 MAR 15:00

    SPAN: 01-APR-2020 - 07-APR-2020

    LOCAL: WEDNESDAY, 01 APR 07:00 - WEDNESDAY, 01 APR 17:00
    UTC: WEDNESDAY, 01 APR 05:00 - WEDNESDAY, 01 APR 15:00

    LOCAL: THURSDAY, 02 APR 07:00 - THURSDAY, 02 APR 17:00
    UTC: THURSDAY, 02 APR 05:00 - THURSDAY, 02 APR 15:00

    LOCAL: FRIDAY, 03 APR 07:00 - FRIDAY, 03 APR 17:00
    UTC: FRIDAY, 03 APR 05:00 - FRIDAY, 03 APR 15:00

    LOCAL: SATURDAY, 04 APR 07:00 - SATURDAY, 04 APR 17:00
    UTC: SATURDAY, 04 APR 05:00 - SATURDAY, 04 APR 15:00

    LOCAL: SUNDAY, 05 APR 07:00 - SUNDAY, 05 APR 17:00
    UTC: SUNDAY, 05 APR 05:00 - SUNDAY, 05 APR 15:00

    LOCAL: MONDAY, 06 APR 07:00 - MONDAY, 06 APR 17:00
    UTC: MONDAY, 06 APR 05:00 - MONDAY, 06 APR 15:00

    LOCAL: TUESDAY, 07 APR 07:00 - TUESDAY, 07 APR 17:00
    UTC: TUESDAY, 07 APR 05:00 - TUESDAY, 07 APR 15:00

    SPAN: 08-APR-2020 - 10-APR-2020

    LOCAL: WEDNESDAY, 08 APR 07:00 - WEDNESDAY, 08 APR 17:00
    UTC: WEDNESDAY, 08 APR 05:00 - WEDNESDAY, 08 APR 15:00

    LOCAL: THURSDAY, 09 APR 07:00 - THURSDAY, 09 APR 17:00
    UTC: THURSDAY, 09 APR 05:00 - THURSDAY, 09 APR 15:00

    LOCAL: FRIDAY, 10 APR 07:00 - FRIDAY, 10 APR 17:00
    UTC: FRIDAY, 10 APR 05:00 - FRIDAY, 10 APR 15:00

    If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

    Best regards,

     

    Team 3W INFRA

     

  • Date - 04/03/2020 07:00 - 10/04/2020 15:00
  • Last Updated - 06/04/2020 16:09
(Re)Scheduled Maintenance Notification - 11 September 2019 (was 3th of September)  (Resolved)
  • Priority - Medium
  • Malfunction Other - Network - NTT
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about the upcoming maintenance by one of our IP transit providers.

    NTT has planned a software upgrade. 

    The maintenance is closely monitored by NTT operations. During this maintenance, connection with NTT Global IP Network (AS2914) will be affected. Traffic shall be rerouted through other available IP transit providers. 3W INFRA engineers will be available during and after the maintenance in case, any issues should arise.

     

    Maintenance Window:

    Start Date and Time: 11 September 2019 02:00 CEST (UTC+2) (was 3th of September) 

    End Date and Time: 11 September 2019 05:00 CEST (UTC+2) (was 3th of September) 

    Action and Reason: Software upgrade.

    Location of Work: Equinix AM7, Kuiperbergweg 13, Amsterdam, the Netherlands

     

    If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

    Best regards,

    Team 3W INFRA

     

    -------------

    UPDATE from NTT at 03:57 CEST (UTC+2)

    Maintenance has been completed without any incident and services have returned to normal. 
    Start Date and Time: 11 September 2019 02:00 CEST (UTC+2) 
    Completion Date and Time: 11 September 2019 03:31 CEST (UTC+2)

  • Date - 11/09/2019 02:00 - 11/09/2019 05:00
  • Last Updated - 26/01/2020 20:11
Scheduled (Emergency) Maintenance Notification - 20 July 2019 (Resolved)
  • Priority - Medium
  • Malfunction Other - Network - NTT
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about the upcoming (emergency) maintenance by one of our IP transit providers.

    NTT has planned a software upgrade. 

    The maintenance is closely monitored by NTT operations. During this maintenance, connection with NTT Global IP Network (AS2914) will be affected. Traffic shall be rerouted through other available IP transit providers. 3W INFRA engineers will be available during and after the maintenance in case, any issues should arise.

     

    Maintenance Window:

    Start Date and Time: 20 July 2019 02:05 CEST (UTC+2)

    End Date and Time: 20 July 2019 05:00 CEST (UTC+2)

    Action and Reason: Software upgrade.

    Location of Work: Equinix AM7, Kuiperbergweg 13, Amsterdam, the Netherlands

     

    If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

    Best regards,

    Team 3W INFRA

  • Date - 20/07/2019 02:05 - 20/07/2019 05:00
  • Last Updated - 26/01/2020 20:09
Scheduled Maintenance Notification - 8 July 2019 (Resolved)
  • Priority - Medium
  • Malfunction Other - Network - Core#1
  • Dear Customer,

    As part of our commitment to continually improve the quality of service, we want to inform you about the upcoming maintenance.

    We have planned an upgrade of part of our core-network equipment and its software. 

    The maintenance is, of course, closely monitored by 3W Infra operations. During this maintenance, no impact on services is to be expected. 3W Infra engineers will be available during and after the maintenance in case, any issues should arise.

    Maintenance Window:

    Start Date and Time: 08 July 2019 05:00 CEST (UTC +2)

    End Date and Time: 09 July 2019 18:00 CEST (UTC +2)

    Action and Reason: Core-network equipment and software upgrade (Core-Router #1 and Core-Switch #1).

    Location of Work: Equinix AM11 (AMS DC-01), Lemelerbergweg 28, Amsterdam, the Netherlands.

    Customer Action Required: We advise all customers with a Private Rack (Colocation) infrastructure to check the configuration of its network equipment to make sure the redundant uplink(s) provided will function properly during the maintenance.

     If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

    Best regards,

    Team 3W INFRA

  • Date - 08/07/2019 05:00 - 09/07/2019 18:00
  • Last Updated - 26/01/2020 20:06
Scheduled Maintenance Notification - 6 December 2018 (Resolved)
  • Priority - Medium
  • Malfunction Other - Network - GTT
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about the upcoming maintenance by one of our IP transit providers.

    GTT has planned a software upgrade. 

    The maintenance is closely monitored by GTT operations (NOC). During this maintenance, no impact to services is to be expected. 3W INFRA engineers will be available during and after the maintenance in case, any issues should arise.

    Maintenance window:
    Start Date and Time: 06 December 2018 02:00 CET (UTC+1)
    End Date and Time: 06 December 2018 06:00 CET (UTC+1)

    Action and Reason: Software upgrade.

    Location of work: Equinix AM7, Kuiperbergweg 13, Amsterdam, the Netherlands

    If you have any questions regarding this maintenance, our engineers are available at support@3winfra.com.

    Team 3W INFRA

  • Date - 06/12/2018 02:00 - 06/12/2018 06:00
  • Last Updated - 26/01/2020 20:03
Scheduled Maintenance Notification - 05 November 2018 (Resolved)
  • Priority - Low
  • Malfunction Other - Switch Datacenters - AMS
  • Dear Customer,

    As part of their commitment to continually improve the quality of service, we want to inform you about the upcoming maintenance by our data center.

    Switch Datacenters has planned an upgrade to their Uninterruptible Power Systems (UPS). 

    The maintenance is, of course, closely monitored by data center operations. During this maintenance, no impact to services is to be expected. 3W INFRA engineers will be available during and after the maintenance in case, any issues should arise.

    Maintenance window:
    Start Date and Time: 05 November 2018 08:00 CET
    End Date and Time: 05 November 2018 18:00 CET

    Action and Reason: Upgrade to Uninterruptible Power Systems (UPS).

    Location of work: Switch Datacenters AMS1 (AMS DC-01), Amsterdam, the Netherlands

    If you have any questions regarding this maintenance, our engineers are available at support at 3winfra.com. 


    Team 3W INFRA

  • Date - 05/11/2018 08:00 - 05/11/2018 18:00
  • Last Updated - 26/01/2020 20:02
RFO 01/16/2020 (Resolved)
  • Priority - Critical
  • Malfunction Other - Network
  • Incident Description:

    At approximately 16/01/2020 23:40 (UTC +1), A high volume of egress network packets are observed from a client's server caused a severe interruption on the core network, during which other clients' BGP connections are disconnected or unstable. Our NOC executed standard mitigation procedure. 

    Mitigation:

    As a temporary measurement, basic port-based filter was applied to the server sending out a high volume of network packets to avoid further impact on the core network. The relevant customer has been informed to have their server inspected.

    Cause:

    The unusual high traffic was suspected to be caused by hacking or a trojan installed on the server.

    Action to take:

    - Maintain a set of network diagram at a secure and accessible location even if the infrastructure is down: Completed

    - Set up proper speed limit and/or QoS rules to prevent normal user traffic affecting BGP sessions: In progress

    - Implement automated traffic monitoring and blocking mechanisms to prevent an abnormal high volume of traffic impacting the core infrastructure: Completed

    - Backup multiple versions of configuration snapshots on the routers and switches automatically to another secure location to prevent accidental changes to configurations: Completed

     

  • Date - 16/01/2020 23:40 - 17/01/2020 02:20
  • Last Updated - 25/01/2020 22:55