[01:53:57] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10155692 (10Papaul) I think replacing the pfw first will be a good idea since we are not changing any configuration on them but ju... [01:56:38] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155693 (10Papaul) [01:57:17] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155694 (10Papaul) [02:13:40] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10155696 (10Papaul) I update the diagram again since we will not be using VC. {F57520229} [02:16:05] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155698 (10Papaul) While working on setting up the new fasw2-c8-codfw I realized that fpc0 has interface ge-0/0/47 connec... [06:12:39] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10155887 (10ABran-WMF) all needed switchover prior to tonight have been done. I'll run T375050 as soon as this is don... [07:39:38] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10155971 (10cmooney) >>! In T373104#10147494, @Jelto wrote: > `gitlab-runner2004` is a special purpose runner, so if w... [07:44:03] 07Puppet, 06Infrastructure-Foundations, 10Keyholder, 06SRE: keyholder-proxy doesn't restart on config change - https://phabricator.wikimedia.org/T374711#10155983 (10elukey) I had a chat with Filippo, the keyholder-proxy is not the daemon that needs re-arming when restarted, so it can be done anytime withou... [08:00:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D3 & D4 from asw to lsw - https://phabricator.wikimedia.org/T373103#10155999 (10cmooney) 05Open→03Resolved a:03cmooney [08:16:15] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10156095 (10ops-monitoring-bot) Draining ganeti2017.codfw.wmnet of running VMs [08:33:51] 10SRE-tools, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10156154 (10MoritzMuehlenhoff) [09:27:43] hey folks, I added some thoughts about the memory usage for puppetservers in https://phabricator.wikimedia.org/T373527#10156328 [09:27:52] if you have time lemme know what you think abut it [09:27:54] *about [09:30:55] makes sense and I'm sure we're in the case where 512MB is not enough ;) [09:32:11] oh yes I am pretty sure we crossed that threshold [09:37:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Support listing pooled / active authdns hosts (rather than all) - https://phabricator.wikimedia.org/T375014#10156356 (10Volans) p:05Triage→03Medium Thanks for the task. I think the main decision to make is how fresh the data needs to be. If we opt f... [12:17:14] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529#10156770 (10cmooney) 05Open→03Resolved Validator is working well to prevent any mis-match, and automation is configuring things correc... [12:35:50] elukey: seems quite reasonable to me [12:36:41] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10156844 (10MoritzMuehlenhoff) ganeti2017 and ganeti2026 are drained [12:42:26] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10156846 (10Jgreen) >>! In T373942#10155516, @Dwisehaupt wrote: > I believe [[ https://phabricator.wikimedia.org/T... [14:07:58] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: cloudgw: add support and enable IPv6 - https://phabricator.wikimedia.org/T374716#10157199 (10aborrero) p:05Triage→03Medium [14:08:06] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10157201 (10aborrero) p:05Triage→03Medium [14:08:14] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: openstack: verify security groups settings for IPv6 - https://phabricator.wikimedia.org/T374714#10157202 (10aborrero) p:05Triage→03Medium [14:08:28] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10157203 (10aborrero) p:05Triage→03Medium [14:58:32] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Support listing pooled / active authdns hosts (rather than all) - https://phabricator.wikimedia.org/T375014#10157485 (10Scott_French) Thanks for taking a look, Riccardo. I should mention, this isn't blocking anything on our end, as I can always do somet... [15:58:43] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157915 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7e878ed4-7126-4f45-87aa-d1087aacf81a) set... [16:06:45] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157936 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9cef1cb8-6d99-4d39-b2db-e242da2fe3f6) set... [16:21:48] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157981 (10cmooney) All hosts moved and responding to ping again. Thanks all for the help! [16:25:53] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10158024 (10ABran-WMF) nodes repooling, haproxy reloaded, thanks for the update @cmooney @Ladsgroup I'll get to T375... [21:27:34] FIRING: DiskSpace: Disk space seaborgium:9100:/ 5.616% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=seaborgium - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:57:35] RESOLVED: DiskSpace: Disk space seaborgium:9100:/ 2.862% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=seaborgium - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace