[08:22:16] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10ema) The issue happened again on cp4025 and a few other nodes. It looks like a deadlock in `librdkafka` to me, the process is spinning on `pthread_cond_wai... [10:00:13] 10netops, 10Operations: No Juniper alarms in SNMP for MX204 - https://phabricator.wikimedia.org/T241105 (10ayounsi) I reached out to our Juniper account rep, after a few emails they opened ER-080949 (Enhancement Request). Since Junos 15, it's possible to generate custom OIDs using python scripts: https://www.... [10:18:52] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10ema) https://github.com/confluentinc/confluent-kafka-go/issues/251 [10:19:15] vgutierrez: hola! If you are online today, can you tell me if https://gerrit.wikimedia.org/r/c/operations/puppet/+/607989 is doable or it is it completly horrible and needs more work? :D [10:19:26] *if it is [10:20:35] sure [10:20:39] I'll check it soon [10:21:08] <3 [10:43:58] elukey: hmmm I wouldn't do it like that [10:44:46] hmm forgive me, it looks good, the diff was messing with my brain [12:26:56] so, it looks like we've got to either patch librdkafka 0.11.6 or backport 1.4.2: https://phabricator.wikimedia.org/T256444#6263718 [12:38:41] vgutierrez: thanks for the review! Will follow your advice about issuing the cert first, didn't think about it [13:47:21] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: cp3053 nvme0 issues - https://phabricator.wikimedia.org/T256632 (10Vgutierrez) [13:47:59] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: cp3053 nvme0 issues - https://phabricator.wikimedia.org/T256632 (10Vgutierrez) p:05Triage→03Medium [13:51:08] XioNoX: it looks like Centurylink, Zayno and Telia love to do maintenance together (all three at the same time on July 2nd) [13:51:50] maybe they're just 1 company now [13:52:36] at least I'm not off on july 2nd :) [13:52:42] :) [13:54:02] 2nd is a lot better than the 3rd [13:55:04] ema: so 2 of those links are the main codfw-eqiad links [13:55:27] we still have other paths, the next one is via eqord [13:56:03] so if they're both down at the same time, it gets uncomfortable but not critical [13:56:46] out of curiosity XioNoX is it possible as a customer to object to a particular time because of such situations? or not really [13:57:51] cdanis: I've seen it happen, depending on the type of maintenance and customer I guess [15:55:56] 10Traffic, 10Operations: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10BBlack) p:05Triage→03Low [15:56:26] ^ just noticed that while looking at the cp3 thing earlier [15:57:19] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10BBlack) [15:58:01] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10BBlack) [16:38:45] 10netops, 10Operations: cr1-codfw:fpc0 failure - https://phabricator.wikimedia.org/T254110 (10RobH) [16:45:49] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10Vgutierrez) we have scheduled a system reboot of these boxes.. I'll sync that with the "re-format" of the NVMe devices. [16:47:02] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10BBlack) [16:57:34] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10elukey) There may be another solution, namely creating a new apt component to hold 1.4.x and deploy it selectively where needed (as opposed to roll it out... [17:51:55] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, and 8 others: Picture from Commons not found from Singapore - https://phabricator.wikimedia.org/T231086 (10Krinkle) [21:17:53] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10wiki_willy) @Jclark-ctr or @Cmjohnson - can one of you doublecheck the s/n's in Netbox? The accounting report says th... [21:23:06] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10ayounsi) It's `TA` from the switches CLI. [21:28:06] 10Traffic, 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Legoktm) >>! In T238803#5680344, @CCicalese_WMF wrote: > As noted in the second last bullet, it is desired that we not archive the ext... [21:34:10] 10Traffic, 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) Makes sense. At this point, I think it makes sense to archive EUCopyrightCampaign and EUCopyrightCampaignSkin. [21:39:47] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10wiki_willy) Cool, thanks @ayounsi. I went ahead and fixed it on the accounting spreadsheet. Thanks, Willy [22:12:51] 10Traffic, 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Legoktm) I filed {T256690} and {T256691}. [22:17:19] 10Traffic, 10Operations, 10Projects-Cleanup, 10Release-Engineering-Team-TODO, and 2 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) Thank you, @Legoktm!