[00:41:39] 10Traffic, 10Operations, 10ops-codfw: cp2018 memory replacement - https://phabricator.wikimedia.org/T191228#4107330 (10Papaul) Hello Papaul, Thank you for replying. We have created the following cases: 1. 7M99F42 - 963059814 – 351594756 (Dispatch) ----- Motherboard and 2 DIMMs 2. 7M6CF42 - 963061588 – 35... [00:47:33] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4107331 (10Papaul) Hello Papaul, Thank you for replying. We have created the following cases: 1. 7M99F42 - 963059814 – 351594756 (Dispatch) ----- Motherboard and 2 DIMMs 2. 7M6CF42 - 963061588 – 35... [01:14:41] 10Traffic, 10Operations, 10ops-codfw: cp2010 memory replacement - https://phabricator.wikimedia.org/T191225#4107347 (10Papaul) Your Service Request SR#: 963061588 Contact Us | Support Library | Download Center | SupportAssist | Community Forums Dear PAPAUL TSHIBAMBA, Current Status: This e-mail serves as c... [01:16:23] 10Traffic, 10Operations, 10ops-codfw: cp2011 memory replacement - https://phabricator.wikimedia.org/T191226#4107348 (10Papaul) Hello Papaul, Thank you for replying. We have created the following cases: 7M99F42 - 963059814 – 351594756 (Dispatch) ----- Motherboard and 2 DIMMs 7M6CF42 - 963061588 – 351604843... [06:44:21] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090#4062522 (10ema) >>! In T190090#4106972, @ayounsi wrote: > About kernel tuning, here are the variables we can adjust as necessary, with their default. > ``` > 50 -- /proc/sys/... [07:11:25] bblack: onboard naming takes precedence, so in lvs2001, eth0 will be named eno1 and not enp3s0f0 [07:19:51] bblack: of course we could (and I think we should) change the default naming policy to get the same kind of naming on all the fleet [07:20:28] so I'd set path naming as the first one, so every NIC would end with interface names like enp3s0f0 [07:23:41] default naming policy --> https://github.com/systemd/systemd/blob/v232/network/99-default.link --> "NamePolicy=kernel database onboard slot path" [07:26:07] onboard > slot > path. Our BIOS currently don't report slot information, so it's not available (except for VMs like pybal-testX), so with the default name policy some NICs will get the onboard name, and others the path name [09:12:38] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10aberjak) Created a ticket in Opera's bug tracking system, internal reference: MT-3735. Thanks for all the info - we will investigate. [09:52:51] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#4108088 (10Iniquity) [10:21:37] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#4108164 (10Deskana) [10:27:01] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#4108187 (10Deskana) [10:37:37] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#4108232 (10Iniquity) [10:44:39] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#4108239 (10Tgr) [13:00:50] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4108532 (10BBlack) Australia experimental results with current peering arrangements: Over 3x serial 24h periods duri... [13:03:46] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4108561 (10BBlack) [13:51:19] bblack: BTW, all smoothness is long gone from https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-caching O:) [14:02:22] vgutierrez: nice, lgtm :) [14:04:35] regarding devices names, like I wrote this morning, onboard takes precedence over path names by default [14:28:41] yeah I saw [14:28:52] honestly, I think we should leave the defaults [14:29:10] we already have deployed stretches anyways, and it would change them, and we don't want lvs to be a one-off different case. [14:29:42] (esp as changing the defaults would change interface naming from installer -> installed without hacking installer settings too) [14:31:00] we already have the lvs interface-mapping stuff laid out per-host anyways. bringing up new LVSes will involve confirming the iface naming and the mapping to rows/switches, but technically it always did, it was just simpler before :) [14:32:47] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10mbaluta) Hi! I'm from Opera Mini server team. We did not move any traffic between DCs on specified dates and I don't see any changes on Mini server si... [14:41:33] 10Traffic, 10Operations, 10ops-codfw: cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T190540#4108873 (10Papaul) @ema @BBlack I need to update BIOS and IDRAC on cp2018 as requested by Dell. Can you please depool the server since it is not sho... [14:41:59] 10Traffic, 10Operations, 10ops-codfw: cp2018 memory replacement - https://phabricator.wikimedia.org/T191228#4108878 (10Papaul) @ema @BBlack I need to update BIOS and IDRAC on cp2018 as requested by Dell. Can you please depool the server since it is not showing depooled on the sheet. Thanks. [15:19:23] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4109027 (10Nuria) Twitter is the best! Thanks for taking time to look into this: @mbaluta: how about we setup a short meeting and we can go over data changes we... [16:28:49] 10Traffic, 10Operations, 10ops-codfw: cp2011 memory replacement - https://phabricator.wikimedia.org/T191226#4109287 (10Papaul) @ema @BBlack I need to update BIOS and IDRAC on cp2011 as requested by Dell. Can you please depool the server since it is not showing depooled on the sheet. Thanks. [16:29:15] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4109289 (10Papaul) @ema @BBlack I need to update BIOS and IDRAC on cp2008 as requested by Dell. Can you please depool the server since it is not showing depooled on the sheet. Thanks. [16:37:59] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4109385 (10RobH) I'm not sure what purpose flashing the bios is going to accomplish. We cannot recreate this error regularly with the current version of the bios, and on other systems we've had this i... [16:43:35] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4109417 (10RobH) Old comment removed due to IRC discussion, Papaul points out they aren't refusing the replacement, but just want updated TSR. cp2008 has been shutdown. it can be flashed and updated... [16:45:49] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4109425 (10Nuria) [16:46:42] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10Nuria) p:05Triage>03High [17:02:09] 10netops, 10Analytics, 10Operations, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#4109537 (10Nuria) 05stalled>03Resolved [17:06:05] 10netops, 10Analytics, 10Operations, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#4109561 (10elukey) Since this task has been open for a long time, I'll open a new one when we'll be ready to create the analytics-in6 filter. [17:11:33] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4109598 (10Papaul) BIOS and IDRAC update complete [17:18:55] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4109636 (10Papaul) TSR upload to Dell [19:03:02] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4110036 (10Nuria) >Normally users from Nigeria connect to a data center in Europe, however between February 17th - March 10th I see a small numb... [20:00:55] 10Traffic, 10Operations, 10ops-codfw: cp2017 memory replacement - https://phabricator.wikimedia.org/T191227#4110136 (10Papaul) The Dell tech call saying he couldn't make it for today. This is now schedule first thing Monday morning . [20:01:09] 10Traffic, 10Operations, 10ops-codfw: cp2010 memory replacement - https://phabricator.wikimedia.org/T191225#4110137 (10Papaul) The Dell tech call saying he couldn't make it for today. This is now schedule first thing Monday morning . [20:01:19] 10Traffic, 10Operations, 10ops-codfw: cp2006 memory replacement - https://phabricator.wikimedia.org/T191223#4110138 (10Papaul) The Dell tech call saying he couldn't make it for today. This is now schedule first thing Monday morning . [20:01:30] 10Traffic, 10Operations, 10ops-codfw: cp2022 memory replacement - https://phabricator.wikimedia.org/T191229#4110139 (10Papaul) The Dell tech call saying he couldn't make it for today. This is now schedule first thing Monday morning . [23:00:04] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4110582 (10Nuria) Varnish5 rollout might have something to do with this? https://gerrit.wikimedia.org/r/#/c/409047/ cc @ema