[00:22:36] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3935923 (10ayounsi) Reviving this thread as T165519 is now resolved. Aiming to do the upgrade on Wednesday Feb 14. All the details from the description should still be accurate. Le... [00:23:45] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3935931 (10ayounsi) [00:54:07] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#3935982 (10BBlack) All of the other lvs10xx will be decommed or spared out to other usage, once these new 4 machines are fully in service and tested. 1x per row makes perfec... [03:40:52] 10Varnish, 10User-Elukey: varnishkafka fails to build on Alpine Linux (strndupa) - https://phabricator.wikimedia.org/T186169#3936093 (10elukey) [07:04:34] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3936304 (10Marostegui) >>! In T172459#3935923, @ayounsi wrote: > Reviving this thread as T165519 is now resolved. > > Aiming to do the upgrade on Wednesday Feb 14. All the details... [08:12:03] 10Varnish, 10User-Elukey: varnishkafka fails to build on Alpine Linux (strndupa) - https://phabricator.wikimedia.org/T186169#3935592 (10MoritzMuehlenhoff) Let's simply use a Debian base image, then? With the overhead that Kafka adds, the disk space saving of musl over glibc is negligable anyway. [11:41:08] 10Traffic, 10Operations, 10Wikimedia-Site-requests: outdated DjVu file page thumbnail in cache - https://phabricator.wikimedia.org/T186153#3937114 (10Aklapper) [12:15:27] 10Traffic, 10Operations, 10Wikimedia-Site-requests: outdated DjVu file page thumbnail in cache - https://phabricator.wikimedia.org/T186153#3935257 (10BBlack) I get the same identifying headers when I query that URL directly from Swift storage internally, indicating this is not an edge caching issue: ``` curl... [12:19:41] 10Varnish, 10User-Elukey: varnishkafka fails to build on Alpine Linux (strndupa) - https://phabricator.wikimedia.org/T186169#3935592 (10BBlack) (also, this may not be the only glibc-specific assumption in the code, it's just one that happened to break compilation. Others could be more-subtle. VK code is fair... [12:28:07] what is the status of eqsin connectivity now? cross connects up yet? [12:29:36] mark: https://phabricator.wikimedia.org/T185657 pending pam for the codfw-side x-connect, at least. [12:29:52] that's pending C level signature now I believe [12:30:28] i'll ping her [12:30:34] thanks! [12:30:46] the SG side is all good according to emails, I think that's all it's waiting on [12:38:53] yup, mail sent [13:30:21] 10Traffic, 10Operations, 10Wikimedia-Site-requests: outdated DjVu file page thumbnail in cache - https://phabricator.wikimedia.org/T186153#3937383 (10Ankry) > I suspect this has more to do with (possibly old) thumbnailing or djvu-handling issues than caching? Newly generated thumbnails of other sizes are ge... [15:46:25] rust is a good cure for jetlag apparently [15:47:06] I've been playing with a few toy programs, and now I've written a prototype for kafkapurge which seems to be working fine [15:47:09] https://phabricator.wikimedia.org/P6650 [15:48:38] disclaimers: I don't necessarily know what I'm doing, whenever the compiler complained about borrowing or similar things I've just done the minimum amount of work required to make it shut up [15:49:20] I'm not sure whether eqiad.mediawiki.job.cdnPurge is the only topic we need to read from [15:49:55] and finally the whole thing is currently single threaded which is likely a bad idea [15:52:19] :) [16:49:14] 10Varnish, 10User-Elukey: varnishkafka fails to build on Alpine Linux (strndupa) - https://phabricator.wikimedia.org/T186169#3938054 (10Jrdnch) Fair enough. Would you consider stating in the documentation that varnishkafka heavily depends on glibc? [16:52:38] 10Varnish, 10User-Elukey: varnishkafka fails to build on Alpine Linux (strndupa) - https://phabricator.wikimedia.org/T186169#3938057 (10elukey) >>! In T186169#3938054, @Jrdnch wrote: > Fair enough. Would you consider stating in the documentation that varnishkafka heavily depends on glibc? We can definitely ad... [17:13:15] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938158 (10ayounsi) @Marostegui do you have an approximate timeline I can base this task on? [17:15:51] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938170 (10Marostegui) >>! In T172459#3938158, @ayounsi wrote: > @Marostegui do you have an approximate timeline I can base this task on? Not really. We'd probably won't be doing... [17:22:31] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938179 (10jcrespo) Maybe your manager should talk to our manager to help us prioritize it? Right now, goals are on our top priority unless said the opposite. Failing over those ser... [17:38:05] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938224 (10BBlack) The question (after the initially proposed 2 week timeline was rejected) was merely "do you have an approximate timeline I can base this task on?". The answer se... [17:47:20] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938249 (10Marostegui) I would definitely vote for waiting to the DC switchover if that is possible and a reasonable timeframe for NetOps. Otherwise, we'd need to squeeze this into... [17:51:02] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3938253 (10Marostegui) [19:13:32] 10Traffic, 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Lower varnish caching length on doc.wikimedia.org - https://phabricator.wikimedia.org/T184255#3877424 (10hashar) The patch got merged and deployed. So tentatively that is fixed? [19:13:41] 10Traffic, 10Continuous-Integration-Infrastructure, 10Operations: Lower varnish caching length on doc.wikimedia.org - https://phabricator.wikimedia.org/T184255#3938632 (10hashar) [19:14:57] 10netops, 10DC-Ops, 10Operations, 10ops-esams: cr2-esams temperature warning - https://phabricator.wikimedia.org/T176816#3938634 (10ayounsi) 05Open>03Resolved a:03mark I haven't seen any alerts since. Thanks. [19:17:12] 10Varnish: latest varnishkafka fails to build on Debian - https://phabricator.wikimedia.org/T186250#3938638 (10Jrdnch) [19:17:52] 10Varnish: latest varnishkafka fails to build on Debian - https://phabricator.wikimedia.org/T186250#3938652 (10Jrdnch) [20:53:32] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459#3939141 (10jcrespo) @BBlack The thing is, we physically could do this in 2 weeks- if we put it on our top priority and do nothing else- I don't know hour urgent is this- if it is lo... [21:41:42] 10Domains, 10Traffic, 10Operations, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3939329 (10Dzahn) p:05Triage>03Normal [22:26:33] 10Domains, 10Traffic, 10DNS, 10Operations, and 2 others: en.wiki domain owned by us, but isn't hosted by us?? - https://phabricator.wikimedia.org/T167060#3939435 (10CRoslof) 05Open>03Resolved a:03CRoslof I've changed the domains we have with Porkbun to use our nameservers. (I didn't change anything a... [23:41:49] ^ ^ :) [23:41:59] or not, eh, i'm not sure [23:42:23] whether we should blackhole it specifically or not [23:49:14] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3920199 (10Dzahn) @Cameron11598 Could you forward Brandon's answer above to the original reporter? Let us know if t... [23:49:29] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3939680 (10Dzahn) p:05Triage>03Normal