[03:28:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:28:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:34:15] FIRING: ProbeDown: Service idp1004:443 has failed probes (http_idp_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:39:15] RESOLVED: ProbeDown: Service idp1004:443 has failed probes (http_idp_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:25:23] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10903091 (10Vgutierrez) p:05Low→03Medium [08:34:05] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10903110 (10Vgutierrez) all PoPs have been running liberica with IPIP encapsulation for a while now and we are already starting the migration to Katran there, so we c... [11:10:38] Hello. We have a couple of servers that we renamed, but we need to move the vlans from analytics to private. The hosts are dse-k8s-worker10[12-13] [11:11:07] Are you happy for us to go ahead with this procedure to do this? https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Old_procedure_(and_for_changing_vlan_if_you_really_need_it) [11:12:50] Or would you rather do it? [11:17:57] why the old manual procedure? [11:20:08] Because it's what we had to use last time, which was only a month or so ago. Happy to use a new procedure it it will work. [11:20:41] The servers were successfully renamed already, using the cookbook. [11:21:13] They were an-db100[1-2] and the new names are dse-k8s-worker101[2-3] [11:21:48] from analytics to private not sure, I would have to check if there are other options, I'll leave it to the vlan experts ;) [11:21:53] sorry lunch ready, bbiab [11:28:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [11:29:09] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (91.32%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [12:40:01] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904311 (10ayounsi) [12:46:45] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904328 (10ayounsi) [12:54:11] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904369 (10ayounsi) [12:57:40] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904383 (10ayounsi) [13:03:52] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904444 (10ayounsi) [13:04:52] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10904451 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [14:27:13] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: [infra] Reports of slow connectivity from APAC - https://phabricator.wikimedia.org/T395135#10904952 (10fnegri) [14:39:13] volans: I'll go ahead with that vlan move using the old method, if that's OK with you. [14:39:32] I don't have context on the previous iteration but if that worked out I gues it's ok [14:53:10] Cool. Thanks. [15:28:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:29:10] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (92.29%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [17:47:19] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10906005 (10cmooney) Nice work @ayounsi! I would have thought dc-ops need to follow up with remote hands requests to remove the cables though? [18:08:25] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: SSD firmware update not working in firmware cookbook - https://phabricator.wikimedia.org/T394543#10906122 (10RobH) 05Open→03Resolved Thanks! [19:28:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [19:29:10] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (93.97%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [20:44:35] 10netops, 06Infrastructure-Foundations, 06Traffic: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#10906579 (10ayounsi) Yep, I opened those too: {T396603} {T396602} {T396601} [23:28:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:29:10] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.23%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure