[01:53:57] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10155692 (10Papaul) I think replacing the pfw first will be a good idea since we are not changing any configuration on them but ju... [01:56:38] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155693 (10Papaul) [01:57:17] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155694 (10Papaul) [02:13:40] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10155696 (10Papaul) I update the diagram again since we will not be using VC. {F57520229} [02:16:05] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10155698 (10Papaul) While working on setting up the new fasw2-c8-codfw I realized that fpc0 has interface ge-0/0/47 connec... [06:12:39] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10155887 (10ABran-WMF) all needed switchover prior to tonight have been done. I'll run T375050 as soon as this is don... [07:12:43] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10155908 (10Vgutierrez) Answering here @RobH question: >Hey I made some assumptions on the cp hosts troubleshooting but should check with you: Those hosts are under the same weight conditions as al... [07:39:38] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10155971 (10cmooney) >>! In T373104#10147494, @Jelto wrote: > `gitlab-runner2004` is a special purpose runner, so if w... [08:00:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D3 & D4 from asw to lsw - https://phabricator.wikimedia.org/T373103#10155999 (10cmooney) 05Open→03Resolved a:03cmooney [08:16:15] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10156095 (10ops-monitoring-bot) Draining ganeti2017.codfw.wmnet of running VMs [08:55:12] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10156264 (10Jelto) In `wikimedia-gitlab`, there have been some reports of failing jobs (cc... [09:52:09] 06Traffic: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10156396 (10Vgutierrez) a:03Vgutierrez [09:52:48] 06Traffic: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10156392 (10Vgutierrez) p:05Triage→03Medium [10:36:27] 06Traffic, 10MW-on-K8s, 06serviceops: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10156467 (10Vgutierrez) a:05Vgutierrez→03None Provided URLs are currently handled by mw-web: ` vgutierrez@carrot:/tmp$ ./T374997.sh https://donate.wikimedia.org/favicon.ico < se... [10:45:11] <_joe_> hi :) [10:46:02] <_joe_> https://phabricator.wikimedia.org/T368654#10143463 can someone explain to me how can a URL that reports cache-control: private, s-maxage=0, max-age=0, must-revalidate is cached at the edge? [10:46:07] <_joe_> am I missing something? [10:50:07] * vgutierrez looking [10:53:25] _joe_: < cache-control: public, s-maxage=3600, max-age=3600 [10:53:46] <_joe_> uhh [10:53:52] <_joe_> see my previous comment :D [10:53:53] vgutierrez@cp6016:~$ curl -v -o /dev/null --connect-to www.wikidata.org:443:$(dig +short mw-web.discovery.wmnet):4450 'https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266' 2>&1 |grep -i cache-control [10:53:53] < cache-control: public, s-maxage=3600, max-age=3600 [10:54:09] the CDN gets s-maxage=3600 [10:54:16] <_joe_> ah yeah no doubt about that now heh [10:54:29] <_joe_> I guess someone botched something in a change :P [10:54:38] <_joe_> and I got the unlucky moment [10:54:39] <_joe_> thanks [11:03:34] let me know if you need me to answer in that task BTW [12:17:14] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529#10156770 (10cmooney) 05Open→03Resolved Validator is working well to prevent any mis-match, and automation is configuring things correc... [12:36:41] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10156844 (10MoritzMuehlenhoff) ganeti2017 and ganeti2026 are drained [12:42:26] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10156846 (10Jgreen) >>! In T373942#10155516, @Dwisehaupt wrote: > I believe [[ https://phabricator.wikimedia.org/T... [14:07:58] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: cloudgw: add support and enable IPv6 - https://phabricator.wikimedia.org/T374716#10157199 (10aborrero) p:05Triage→03Medium [14:08:06] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10157201 (10aborrero) p:05Triage→03Medium [14:08:16] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: openstack: verify security groups settings for IPv6 - https://phabricator.wikimedia.org/T374714#10157202 (10aborrero) p:05Triage→03Medium [14:08:30] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10157203 (10aborrero) p:05Triage→03Medium [14:51:17] 06Traffic, 06cloud-services-team, 06SRE, 13Patch-For-Review: Rename references of labweb to cloudweb - https://phabricator.wikimedia.org/T317463#10157462 (10joanna_borun) p:05Triage→03Low [14:53:00] 06Traffic, 06cloud-services-team, 06SRE, 13Patch-For-Review: Rename references of labweb to cloudweb - https://phabricator.wikimedia.org/T317463#10157468 (10dcaro) Still some stuff to be changed: https://codesearch.wmcloud.org/search/?q=labweb [15:58:43] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157915 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7e878ed4-7126-4f45-87aa-d1087aacf81a) set... [16:06:45] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157936 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9cef1cb8-6d99-4d39-b2db-e242da2fe3f6) set... [16:21:48] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10157981 (10cmooney) All hosts moved and responding to ping again. Thanks all for the help! [16:25:53] 10netops, 06collaboration-services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10158024 (10ABran-WMF) nodes repooling, haproxy reloaded, thanks for the update @cmooney @Ladsgroup I'll get to T375... [18:06:17] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10158427 (10RobH) So the SEL/idrac logs show no thermal events, and dell support is attempting to deny these support requests. On checking cp3071, I don't see any thermal events in the logs: ` r...