Re: [URGENT] : PHX DC is down

Hello. Today we have found that the infrastructure of oVirt, Kubevirt projects running at PHX datacenter is not accessible. It is not accessible from both Red Hat networks and from the internet. Looks to be network related as I did not manage to access anything there. Few main hosts affected are jenkins.ovirt.org (66.187.230.92) and resources.ovirt.org (66.187.230.28). Here is also traceroute to jenkins from internet: # traceroute jenkins.ovirt.org traceroute to jenkins.ovirt.org (66.187.230.92), 30 hops max, 60 byte packets 1 _gateway (10.0.0.138) 11.635 ms 11.614 ms 11.554 ms 2 bzq-179-37-1.cust.bezeqint.net (212.179.37.1) 15.719 ms 17.011 ms 17.012 ms 3 * * * 4 bzq-25-77-10.cust.bezeqint.net (212.25.77.10) 16.951 ms 17.556 ms 17.545 ms 5 bzq-219-189-2.cablep.bezeqint.net (62.219.189.2) 18.053 ms 18.044 ms 73.674 ms 6 bzq-179-124-62.cust.bezeqint.net (212.179.124.62) 85.149 ms * bzq-219-189-73.dsl.bezeqint.net (62.219.189.73) 77.685 ms 7 bzq-179-72-242.cust.bezeqint.net (212.179.72.242) 70.555 ms ge-2-1-0.mpr1.lhr2.uk.above.net (195.66.224.76) 75.340 ms ae11.mpr2.lhr2.uk.zip.zayo.com (64.125.30.52) 70.436 ms 8 ge-2-1-0.mpr1.lhr2.uk.above.net (195.66.224.76) 76.982 ms ae11.mpr2.lhr2.uk.zip.zayo.com (64.125.30.52) 76.960 ms ge-2-1-0.mpr1.lhr2.uk.above.net (195.66.224.76) 71.310 ms 9 * * * 10 * * * 11 ae0.cs2.lga5.us.eth.zayo.com (64.125.29.187) 208.885 ms * ae4.cs2.dca2.us.eth.zayo.com (64.125.29.31) 195.693 ms 12 ae4.cs2.dca2.us.eth.zayo.com (64.125.29.31) 232.666 ms 200.411 ms 212.885 ms 13 ae26.mpr4.phx2.us.zip.zayo.com (64.125.31.255) 200.021 ms ae3.cs2.iah1.us.eth.zayo.com (64.125.29.45) 200.916 ms ae4.cs2.dca2.us.eth.zayo.com (64.125.29.31) 196.332 ms 14 ae26.mpr4.phx2.us.zip.zayo.com (64.125.31.255) 206.193 ms 206.258 ms ae3.cs2.iah1.us.eth.zayo.com (64.125.29.45) 197.296 ms 15 ae0.mpr3.phx2.us.zip.zayo.com (64.125.27.12) 207.483 ms ae26.mpr4.phx2.us.zip.zayo.com (64.125.31.255) 202.373 ms ae0.mpr3.phx2.us.zip.zayo.com (64.125.27.12) 208.078 ms 16 ae0.mpr3.phx2.us.zip.zayo.com (64.125.27.12) 204.214 ms border5.ae2-bbnet2.phx010.pnap.net (69.25.168.65) 200.320 ms 208.184.108.38.IPYX-072053-007-ZYO.above.net (208.184.108.38) 203.315 ms 17 redhat-2.edge1.phx004.pnap.net (69.25.121.26) 200.961 ms 208.184.108.38.IPYX-072053-007-ZYO.above.net (208.184.108.38) 198.863 ms border5.ae2-bbnet2.phx010.pnap.net (69.25.168.65) 206.668 ms 18 redhat-2.edge1.phx004.pnap.net (69.25.121.26) 206.648 ms unused (66.187.228.249) 227.743 ms unused (66.187.228.248) 228.747 ms 19 redhatphx-1.phx004.pnap.net (69.25.120.242) 203.819 ms redhat-2.edge1.phx004.pnap.net (69.25.121.26) 199.135 ms * 20 * unused (66.187.228.248) 241.699 ms * 21 * * * 22 * * * The services running there such as jenkins are business critical for oVirt (and RHV) development. Also resources is used to publish oVirt externally.