[06:35:13] 10Traffic, 10Analytics, 10Operations: varnishkafka statsv and webrequest crashed on cp1081 - https://phabricator.wikimedia.org/T231331 (10elukey) I agree with Andrew, the issue seems to be a violation of an assert or similar in the Varnish libs, so unlikely related to a Varnishkafka bug (famous last words).... [07:47:15] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review: ATS Backends: Test live cache_text traffic - https://phabricator.wikimedia.org/T228629 (10ema) 05Open→03Resolved cp1075 has been serving live production traffic for several days now, we can consider the test successful. [07:47:19] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ema) [07:50:35] ema, vgutierrez: the "traffic_server tls process restarted" alert on cp5001 jus triggered (2 ge 2) [07:50:57] im aware thx [07:51:09] I'll take care of that on a few minutes [08:12:50] 10Traffic, 10Operations: Investigate segfaults on ats-tls running on cp5001 - https://phabricator.wikimedia.org/T232298 (10Vgutierrez) [08:26:44] 10Traffic, 10Operations, 10Patch-For-Review: Investigate segfaults on ats-tls running on cp5001 - https://phabricator.wikimedia.org/T232298 (10Vgutierrez) Both crashes seems to be related: `name=crash-2019-09-05-155443.log Thread 12661, [ET_NET 32]: 0 0x000000000049d9f0 crash_logger_invoke(int, siginfo_t*... [09:08:36] 10Traffic, 10Operations, 10Patch-For-Review: Investigate segfaults on ats-tls running on cp5001 - https://phabricator.wikimedia.org/T232298 (10Vgutierrez) service log shows memory issues: ` Sep 05 15:54:43 cp5001 traffic_server[12607]: Fatal: couldn't allocate 32768 bytes Sep 08 03:24:10 cp5001 traffic_serve... [09:08:46] 10Traffic, 10Operations, 10Patch-For-Review: Investigate segfaults on ats-tls running on cp5001 - https://phabricator.wikimedia.org/T232298 (10Vgutierrez) p:05Triage→03Normal [09:37:53] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service: LDF service does not Vary responses by Content-Type, sending incorrect cached responses to clients - https://phabricator.wikimedia.org/T232006 (10jbond) p:05Triage→03Normal [10:05:47] 10Traffic, 10Operations: PyBal ProxyFetch checks using HTTP/1.0 with https and HTTP/1.1 with plain http - https://phabricator.wikimedia.org/T232319 (10ema) [10:06:41] 10Traffic, 10Operations: PyBal ProxyFetch checks using HTTP/1.0 with https and HTTP/1.1 with plain http - https://phabricator.wikimedia.org/T232319 (10ema) p:05Triage→03Normal [12:55:33] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [12:59:49] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:03:50] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:26:18] paladox: https://gerrit.wikimedia.org/r/c/operations/puppet/+/535184 [13:26:43] vgutierrez thanks! [13:34:26] new cert pushed paladox [13:34:31] willikins:~ vgutierrez$ openssl s_client -connect gerrit.wikimedia.org:443 /dev/null |openssl x509 -noout -text|grep "DNS:" [13:34:31] DNS:gerrit-replica.wikimedia.org, DNS:gerrit.wikimedia.org [13:34:41] thanks! :) [13:34:49] the replica will get it as soon as puppet runs there [13:51:16] 10Traffic, 10Accuracy-Review-of-Wikipedias, 10Bad-Words-Detection-System, 10Better Use Of Data, and 88 others: Deprecate jquery.throttle-debounce in favour of OO.ui.debounce/throttle - https://phabricator.wikimedia.org/T213426 (10GoogleLegacy) [14:57:27] hey! [14:57:48] this https://gerrit.wikimedia.org/r/c/operations/puppet/+/529053 could use some traffic love :) [14:59:44] I guess that I'm not part of traffic then /o\ [15:00:25] onimisionipe: E_TOOLATE here already, but if you need to merge it, we can merge it tomorrow if that works for you [15:01:10] vgutierrez: you are :) and yes, tomorrow works just fine! [15:01:16] thanks! [15:08:00] thanks for the work on that cr onimisionipe [15:19:46] onimisionipe: cool :) [15:58:24] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, and 2 others: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) @Cmjohnson / @Jclark-ctr https://gerrit.wikimedia.org/r/535221 adds DNS for non mgmt entries. Sho... [16:15:51] elukey: yw! [19:53:26] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) ping @ema now that ahem, things are a bit more quiet [22:43:39] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 6 others: Picture from Commons not found from Singapore - https://phabricator.wikimedia.org/T231086 (10CDanis) >>! In T231086#5447327, @CDanis wrote: > On each Swift frontend host, I: > * grepped today's logs for GETs that resulted in 404... [23:11:01] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10media-storage: upload LB: retry swift 404s cross-cluster - https://phabricator.wikimedia.org/T231108 (10CDanis) @BBlack @ema can you weigh in at some point soon with how feasible this seems? I'm pretty unfamiliar with the current setup h... [23:15:35] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10media-storage: upload LB: retry swift 404s cross-cluster - https://phabricator.wikimedia.org/T231108 (10BBlack) @ema would know better about how difficult such things are with ATS in particular. I tend not to like this idea in general, t... [23:19:36] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10media-storage: upload LB: retry swift 404s cross-cluster - https://phabricator.wikimedia.org/T231108 (10CDanis) Yeah, all fair points. We don't seem to be experiencing too many of these 404s (a handful per day), and other mitigations are... [23:20:00] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10media-storage: upload LB: retry swift 404s cross-cluster - https://phabricator.wikimedia.org/T231108 (10CDanis) 05Open→03Declined [23:20:07] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 6 others: Picture from Commons not found from Singapore - https://phabricator.wikimedia.org/T231086 (10CDanis)