[08:22:11] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10ema) >>! In T256444#6264956, @elukey wrote: > There may be another solution, namely creating a new apt component to hold 1.4.x and deploy it selectively wh... [08:29:41] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: cp3053 nvme0 issues - https://phabricator.wikimedia.org/T256632 (10Vgutierrez) 05Open→03Stalled repooled after powercycling & issuing the following commands: ` /usr/sbin/nvme format /dev/nvme0n1 -l 2 echo ';' | /usr/sbin/sfdisk /dev/nvme0n1 ` I'll keep... [08:44:50] 10Traffic, 10Operations, 10Patch-For-Review, 10Sustainability (Incident Prevention): monitoring & alerting for purged - https://phabricator.wikimedia.org/T256446 (10ema) [08:54:05] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2027-2028].codfw.wmnet ` [09:40:24] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2029-2030].codfw.wmnet ` [10:06:24] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10elukey) >>! In T256444#6266680, @ema wrote: > Well, [[ https://github.com/edenhill/librdkafka/issues/2020 | upstream claims ]] that the new versions are AP... [10:45:59] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2031-2032].codfw.wmnet ` [12:03:36] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2033-2034].codfw.wmnet ` [13:31:38] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2035-2036].codfw.wmnet ` [16:40:56] speaking of codfw hardware [16:41:20] we still have cp2003, 2009, 2015, and 2021 as 4 nodes of "test cluster" held back from the old codfw hardware [16:41:39] the comment say they were spares in case we needed to backfill dead hardware before the new stuff arrived [16:41:50] I think at this point we can decom and release them [17:44:21] 10Traffic, 10Core Platform Team, 10Operations, 10serviceops, and 3 others: Reduce rate of purges emitted by MediaWiki - https://phabricator.wikimedia.org/T250205 (10aaron) >>! In T250205#6158994, @Krinkle wrote: >>>! In T250205#6154883, @aaron wrote: >> I'm not fond of the idea of not sending purges for in... [17:56:45] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) [18:12:55] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10Cmjohnson) [18:14:11] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10Cmjohnson) new-msw1-eqiad has the correct JUNOS 18.1.3 and the configuration has been copied. Currently connected to port 2 on the a8-scs and can be moved to th... [18:31:04] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) Okay, here are some backtraces: {P11715} When I saw crashes in malloc and then installed libc6-dbg to get arguments, I was hoping that the issue was malloc being invoked with a ridiculous paramet... [18:34:21] 10Traffic, 10Operations: Switch blog.wikimedia.org to diff.wikimedia.org - https://phabricator.wikimedia.org/T254367 (10CKoerner_WMF) [18:34:51] 10Traffic, 10Operations: Switch blog.wikimedia.org to diff.wikimedia.org - https://phabricator.wikimedia.org/T254367 (10CKoerner_WMF) Update, as probably obvious, we have pushed the launch date back. Our target date is now July 14th. [18:35:14] 10Traffic, 10Operations: Switch blog.wikimedia.org to diff.wikimedia.org - https://phabricator.wikimedia.org/T254367 (10Dzahn) [18:39:02] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) {P11716} [18:52:01] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) Filed upstream https://github.com/pmacct/pmacct/issues/414 [20:27:47] 10Traffic, 10Operations: varnishmtail panics on buster - https://phabricator.wikimedia.org/T243591 (10colewhite) [21:07:54] 10Traffic, 10Operations: Certain links being rejected by caching if opened in Internet Explorer - https://phabricator.wikimedia.org/T256302 (10Aklapper) @Urbanecm: What is the exact HTTP error type? Asking as that screenshot does not include any error message (probably not to expose the IP). At least on the "...