[05:45:22] 10netops, 10Operations, 10monitoring: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144#3434183 (10ayounsi) Test instance upgraded to the latest master. [06:51:36] 10netops, 10Operations: Remove unsecure SSH algorithms on network devices - https://phabricator.wikimedia.org/T170369#3434238 (10ayounsi) 05Open>03Resolved Pushed to all network devices. We can revisit the automated scans at a later date (or a common network/systems goal). [07:24:36] 10Traffic, 10Operations: Non zero rated LVS IPs - https://phabricator.wikimedia.org/T170518#3434273 (10ayounsi) [07:58:03] 10Traffic, 10Operations, 10monitoring, 10Patch-For-Review: Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy - https://phabricator.wikimedia.org/T161101#3434317 (10ema) 05Open>03Resolved a:03ema >>! In T161101#3431603, @faidon wrote: > @Ema, it seems like the ta... [12:22:53] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3434956 (10Rap2h) [12:23:40] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3434969 (10Rap2h) [12:41:53] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3434956 (10Urbanecm) Hi, all logos should be already optimalized by OptiPNG (the command recommended for optimalization is `optipng -o7 `). It is up to discussion if ot... [12:42:26] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435052 (10Urbanecm) p:05Triage>03Lowest [13:03:46] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435139 (10Rap2h) Ok. I'm not sure to understand what you suggest me to do. Can I open a pull request on this Github repo after optimizing all images? Or should I follow this tutorial:... [13:12:23] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3434956 (10BBlack) I think the main point here is we'd rather have a reproducible method for optimizing these images which works on our Linux and open-source based infrastructure. Havi... [13:14:24] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435191 (10Framawiki) [13:18:33] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435217 (10Urbanecm) >>! In T170546#3435139, @Rap2h wrote: > Ok. I'm not sure to understand what you suggest me to do. Can I open a pull request on this Github repo after optimizing all... [13:22:24] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435228 (10Framawiki) (Priority levels corresponds to the urgency of the task and the speed required, particularly in #wikimedia-site-requests) [13:40:16] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435376 (10Rap2h) Ok thanks to all! (I'm still discovering the contribution rules, so thanks to help and mentor me!) > can you document a process we could use to apply this universall... [13:59:17] 10Traffic, 10Operations: Support TLSv1.3 - https://phabricator.wikimedia.org/T170567#3435393 (10BBlack) [14:05:47] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3435430 (10elukey) A spike just happened and one interesting thing that I noticed is that the varnishkafka producers seems not affected: https://grafana.wikimedia.org/dashboard/db/varn... [14:07:52] 10Traffic, 10Operations, 10Wikimedia-Site-requests: Optimize Wikipedia PNG Logo - https://phabricator.wikimedia.org/T170546#3435445 (10Urbanecm) Well, for developing a process you aren't required to know what the tool does in the background (although it's better to know than don't know). When I decide to upd... [14:21:29] so I just checked on cp4017 and graphite seems missing some metrics [14:31:45] weird, it says WARNING Failed to get lock. Is another instance of logster running? [14:33:24] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3435667 (10Fjalapeno) [14:34:03] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#2231032 (10Fjalapeno) @tgr is this still valid? [14:40:21] ah, removed the locks in /var/run/*logster* and it worked [14:40:38] now I need to figure out how many of them are not working [14:43:54] sigh I see the fetch failed persist + 500s persist heh [14:45:50] I need to investigate what is logster doing [15:01:05] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3435845 (10Tgr) It's blocked on the two open subtasks, T164790 and T164791 (arguably the latter is less important and could be done post-deploy as... [15:32:11] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3436015 (10dr0ptp4kt) I'm in favor of deployment to wikitech for starters, where the highly technical crowd can handle the complexity. @Deskana @j... [15:45:58] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3436066 (10Tgr) I believe templates in Flow comments/summaries would be broken in both edit and read mode. But then, complex templates are almost... [15:49:11] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3436107 (10elukey) Re-adding the comment after checking that all the cp hosts were pushing metrics to graphite (on some of them logster was erroring for lock not acquired). From https:... [16:35:51] 10Traffic, 10Operations: Extending our HSTS value beyond ~1y - https://phabricator.wikimedia.org/T170598#3436309 (10BBlack) [19:21:21] 10netops, 10Operations: pfw-codfw still logging to indium - https://phabricator.wikimedia.org/T170622#3437338 (10Jgreen) [20:05:00] 10Traffic, 10Operations, 10media-storage: HTTP 429 on image on Commons - https://phabricator.wikimedia.org/T170628#3437497 (10Superzerocool) [20:06:28] 10Traffic, 10Operations, 10media-storage: HTTP 429 on image on Commons - https://phabricator.wikimedia.org/T170628#3437515 (10Superzerocool) [21:17:46] 10Traffic, 10Operations, 10media-storage: HTTP 429 on image on Commons - https://phabricator.wikimedia.org/T170628#3437497 (10Zoranzoki21) I too. {F8738207} {F8738208} {F8738209} [22:07:33] 10Traffic, 10Commons, 10Operations, 10media-storage: HTTP 429 on image on Commons - https://phabricator.wikimedia.org/T170628#3437878 (10Framawiki) Possibly related to {T170605}, the error is different [22:07:50] 10Traffic, 10Commons, 10Operations, 10media-storage: ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION - https://phabricator.wikimedia.org/T170605#3436479 (10Framawiki) [22:08:04] 10Traffic, 10Commons, 10Operations, 10media-storage: HTTP 429 on image on Commons - https://phabricator.wikimedia.org/T170628#3437497 (10BBlack) So, error code 429 is `Too Many Requests`, generally used by ratelimiters. In this case, it seems that thumbor (our internal service that renders thumbnails of i...