[09:17:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Servers on public1-a-codfw and public1-b-codfw not getting DHCP during reimage - https://phabricator.wikimedia.org/T358488#9579500 (10cmooney) 05Open→03Resolved Patch tested again and still working consistently, I think the initial prob... [11:14:36] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9579883 (10akosiaris) [11:16:32] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9537132 (10akosiaris) The LVS traffic approach was doomed to fail, since scap utilizes the same data s... [11:52:21] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9580018 (10akosiaris) [11:52:41] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9537132 (10akosiaris) Migration started, we are batch 1 for the next few days. [12:09:50] 10Traffic, 10DC-Ops, 10Data-Persistence, 10collaboration-services, and 4 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9543213 (10LSobanski) [14:24:19] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580452 (10akosiaris) [14:24:34] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580465 (10akosiaris) p:05Triage→03High [14:33:58] 10netops, 10Infrastructure-Foundations, 10SRE: Servers on public1-a-codfw and public1-b-codfw not getting DHCP during reimage - https://phabricator.wikimedia.org/T358488#9580482 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2004.codfw.wmnet with... [14:34:42] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580484 (10ssastry) It doesn't show up in production because the logging... [14:53:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803#9580655 (10cmooney) >>! In T345803#9479281, @Papaul wrote: > @cmooney can we get those 2 hosts back in decom? Thanks I'm done with sretes... [15:00:28] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803#9580694 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cmooney@cumin1002 for hosts: `sretest2004.codfw.wmnet` - sretes... [15:12:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9580720 (10cmooney) [15:24:39] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9580752 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cmooney@cumin1002 for hosts: `testvm2001.codfw.wmnet` - testvm2001.codf... [15:46:08] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9580810 (10Lucas_Werkmeister_WMDE) [15:55:41] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9580837 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=94e7352f-26c7-48ff-b2c5-61b1faed7b5a) set by cmooney@cumin1002 fo... [15:56:54] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9580845 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=4a16f229-e545-4883-81ab-3b2ddd2d7636) set by cmooney@cumin1002 fo... [16:15:50] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9580894 (10cmooney) All moves complete, everything looking good and back responding to ping :) [16:17:09] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9580897 (10ABran-WMF) thanks! will repool! [21:41:07] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9581849 (10Izno) {T200517} has some work done on it that could maybe chan... [22:50:38] 06Traffic, 06DC-Ops, 06SRE, 10ops-eqiad: Decommission task for old cp hosts (cp1075-1090) - https://phabricator.wikimedia.org/T352253#9581998 (10dr0ptp4kt) After setup, I would be interested in using it for 6 weeks if that's okay (hopefully things would only take 4 weeks, but there's some PTO and real life...