[11:10:55] 10Traffic, 10Operations: Renew GlobalSign Unified in 2018 - https://phabricator.wikimedia.org/T206804 (10BBlack) Also remembering there's some stats @Krinkle mentioned in T196248 about clock skews and users from some Google research. The TL;DR there was 24 hours gives us 93.3% , and 5 days is the sweet spot g... [11:11:34] bblack, wondering if we should have a phab project for certcentral [11:15:16] 10Traffic, 10Operations: Renew GlobalSign Unified in 2018 - https://phabricator.wikimedia.org/T206804 (10BBlack) [12:44:04] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10Jgreen) >>! In T188561#4656191, @CCogdill_WMF wrote: > @Jgreen @BBlack thanks for bumping this and continuing to check. It does look like the SS... [14:32:26] 10Traffic, 10Operations: 1 power feed down in eqsin - https://phabricator.wikimedia.org/T206861 (10ayounsi) p:05Triage>03Normal [14:33:03] 10Traffic, 10Operations: 1 power feed down in eqsin - https://phabricator.wikimedia.org/T206861 (10ayounsi) [14:34:40] bblack: we depooled eqsin: https://phabricator.wikimedia.org/T206861 [14:35:31] not sure if we want to repool during the weekend [14:37:45] yeah I see, thanks [14:38:06] the biggest problem is that we didn't notice the maintenance announce earlier [14:38:14] but that aside: when does their maint window end? [14:38:27] looks like +6h from now? [14:38:35] roughly [14:39:09] bblack: 10h [14:39:22] crazy [14:39:30] on a friday night [14:40:54] I see: [14:40:56] UTC:FRIDAY, 12 OCT 14:00 - SATURDAY, 13 OCT 00:00 [14:41:25] so yeah, 10h window starting ~40 mins ago [14:41:52] that puts it... ~7pm my time when the window ends. [14:43:11] I'll be busy around that time with a kid's softball game, but I can check back in around maybe 9pm (or somewhere thereabout) and make sure everything's been back on a while and no new emails from equinix about extensions/risks, and repool then. [14:43:20] better than leaving the latency high all weekend. [14:44:30] or maybe we go much sooner, if they end up finishing this quickly and announcing an early end to the window, we'll see [14:44:37] indeed [14:44:55] I can have a look saturday morning my time too (UTC+1) [14:48:13] ok [17:40:23] 10Domains, 10Traffic, 10Operations: SOA serial numbers returned by authoritative nameservers differ - https://phabricator.wikimedia.org/T206688 (10Dzahn) I have replied by email and referred to this ticket. I also pointed out we don't agree that this check is even useful (but that it was fixed nevertheless).... [17:41:04] 10Domains, 10Traffic, 10Operations: SOA serial numbers returned by authoritative nameservers differ - https://phabricator.wikimedia.org/T206688 (10Dzahn) 05Open>03Resolved Thanks again. As always it's greatly appreciated that you add all the technical details and background. [18:04:44] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances1-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) p:05Triage>03High [18:06:20] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10aborrero) [18:06:38] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) So the vlans show: ``` default-switch cloud-instances1-b-eqiad 1102... [18:12:33] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) So to show the output: ``` robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1023 ge-1/0/8 up u... [18:15:23] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10Papaul) ``` papaul@asw2-b-eqiad# ...eqiad unit 0 family ethernet-switching vlan members cloud- {master:2}[edit] papaul@asw2... [18:20:24] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) So, the above from @papaul would add in the vlan a second time, since hte current config already shows: ``` interface-ra... [18:34:14] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) >>! In T206872#4661995, @Papaul wrote: > ``` > papaul@asw2-b-eqiad# ...eqiad unit 0 family ethernet-switching vlan member... [19:37:37] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10faidon) 05Open>03Resolved a:03faidon FWIW, this was sorted out for both cloudvirt1023 and cloudvirt1024 in exactly the way... [19:38:21] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10Andrew) thanks all! [19:42:38] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10Andrew) Would you expect us to need to reinstall the OS (or otherwise make interface changes) after this change? [19:53:59] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10RobH) I wouldn't think so, nope. The OS install doesn't really make use of eth1... [20:11:38] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10aborrero) 05Resolved>03Open This still doesn't work. Same behavior as before: packets go out of the interface but nothing en... [20:13:42] 10Traffic, 10Operations: certcentral: check for SCTs, with optional disable per-account - https://phabricator.wikimedia.org/T206876 (10BBlack) p:05Triage>03Normal [20:14:01] bblack, so what did you think about a phab project? [20:14:30] Krenair: probably makes sense! [20:24:11] ok, will get that done [20:25:20] https://phabricator.wikimedia.org/T206877 [20:56:07] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10Papaul) I checked the other cloudvirt node that are working (1021,1020 and 1022) all of their eth1 is part of the interface-rang... [23:33:11] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan - https://phabricator.wikimedia.org/T206872 (10faidon) 05Open>03Resolved >>! In T206872#4662465, @Papaul wrote: > I checked the other cloudvirt node that are working (1021...