[06:06:15] V10000000 [06:06:29] 100000 [06:06:44] Guest38: 00000 [07:48:45] oO [07:52:38] Amir1: is the schema change in codfw s4 still running? If not can I start my schema change? [08:36:35] Amir1: as a temporary workaround I deleted the stale file on es2049 to clear the alarm - the partinioning is ok so we can pool the server in and move on to the next server [13:16:50] Sorry I missed pings here... [13:17:56] federico3: s4 codfw will take until next week to finish. Let it be. It's a slow schema change [13:18:10] I check the es one asap [13:18:39] i can start s4 codfw now and keep an eye on it [13:18:54] ...is that ok? [13:25:57] assuming yes, starting s4 in codfw :) [14:33:42] from what I'm seeing, each categorylinks schema change takes around three hours + the repool time, it'll be four. three more left which means it'll take 12 more hours, while the afl schema change take a couple of minutes so it'll actually reach the categorylinks and cause a royal mess. plus now two replicas have been depooled at the same time. Let's stop afl and repool the host now and pick it up on Monday [14:50:11] ok [14:58:29] Amir1: then maybe I can run T401906 on s5/s6 in eqiad in the meantime? [14:58:29] T401906: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906 [15:22:56] Sounds good [15:34:51] regarding https://gerrit.wikimedia.org/r/c/operations/puppet/+/1184544 tappof is going to be OOtO in the next weeks (and today) and we can either use workarounds like cleaning up the cache file and/or silencing alerts or discuss alternatives. Amir1 I put you in Cc as well if you want to chime in [15:36:34] I would need time to dive into it. I miss a lot of context otherwise I know it's not a lot of work [15:36:36] Those checks aren’t really in production yet (I’m just testing their functionality in production after a first round on the Pontoon stack), so as far as I’m concerned, there’s nothing preventing us from silencing the alerts until I’m back.