[08:53:46] 10Traffic, 10Performance-Team, 10SRE: ats-be occasional system CPU usage increase - https://phabricator.wikimedia.org/T265625 (10ema) Disabling JIT in all Lua scripts resulted in significantly decreased CPU usage [[https://grafana.wikimedia.org/d/7-ZqK8-Wz/varnish-frontend-ttfb-comparison?viewPanel=4&orgId=1... [09:52:09] sukhe: s/^my /the / [10:17:45] hi all fyi im going to disable puppet on the cp::text for a short period to deploy some block list changes 651174 + 651171 [10:18:30] jbond42: sounds good, nice work! [10:19:03] thanks ema [10:30:48] 10HTTPS, 10Traffic, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.beta.wmflabs.org expired on January 12, 2021. - https://phabricator.wikimedia.org/T271808 (10Aklapper) >>! In T271808#6739592, @RhinosF1 wrote: > But yeah I guess this should be fixed/monitored better so it doesn't need manu... [10:34:29] jbond42: puppet re-enabled, right? [10:35:17] yes puppet re-enabled and the agent run just finished so should be deployed everywhere now [10:35:24] excellent! [10:37:33] ema: got an evil plan brewing https://media.giphy.com/media/8fen5LSZcHQ5O/source.gif ;) [10:38:24] mmuahahaha yes [10:38:42] lol :D [10:43:18] 10HTTPS, 10Traffic, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.beta.wmflabs.org expired on January 12, 2021. - https://phabricator.wikimedia.org/T271808 (10RhinosF1) >>! In T271808#6742982, @Aklapper wrote: >>>! In T271808#6739592, @RhinosF1 wrote: >> But yeah I guess this should be fix... [14:46:02] 10Traffic, 10SRE, 10ops-eqiad: Interface errors on asw2-a-eqiad:xe-4/0/7 (lvs1016) - https://phabricator.wikimedia.org/T271087 (10elukey) The interface seems having trouble at the moment, we have some icinga alerts about pybal not reaching row-a hosts: ` elukey@asw2-a-eqiad> show interfaces descriptions | m... [14:47:37] 10Traffic, 10SRE, 10ops-eqiad: Interface errors on asw2-a-eqiad:xe-4/0/7 (lvs1016) - https://phabricator.wikimedia.org/T271087 (10ayounsi) p:05Medium→03High [14:48:00] Can someone from traffic sync up with DCops for https://phabricator.wikimedia.org/T271087 ? [14:54:02] 10Traffic, 10SRE, 10ops-eqiad: Interface errors on asw2-a-eqiad:xe-4/0/7 (lvs1016) - https://phabricator.wikimedia.org/T271087 (10elukey) The lvs is a secondary so not taking traffic, added a day of downtime :) [14:59:11] 10Traffic, 10netops, 10SRE: Create Generalised blocking stratagy - https://phabricator.wikimedia.org/T270618 (10CDanis) Thanks for the writeup with all the background! And for the cleanup patches so far :) Just a few things to add. Re: the schema of blocks themselves: * In addition to being able to specif... [16:22:16] 10Traffic, 10SRE, 10ops-eqiad: lvs1016 interface down - https://phabricator.wikimedia.org/T271087 (10ayounsi) [18:18:34] 10Traffic, 10CheckUser, 10SRE: Log source port for anonymous users and expose it for sysops/checkusers - https://phabricator.wikimedia.org/T181368 (10Ladsgroup) I'm inclined to close this as declined in favor of {T271953} which basically gives people who have access to hadoop to be able to see the source por... [18:34:43] 10HTTPS, 10Traffic, 10SRE, 10serviceops, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10Nintendofan885) [18:35:12] 10HTTPS, 10Traffic, 10Prod-Kubernetes, 10serviceops, and 2 others: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10Nintendofan885) [18:36:18] 10HTTPS, 10Traffic, 10SRE, 10serviceops, and 3 others: Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10Nintendofan885)