[08:35:01] if someone around has 1min to give my patch a sanity check that would be amazing :) [08:35:07] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071883 [08:35:25] (please and thank you <3) [10:55:48] PROBLEM - MariaDB sustained replica lag on s3 on db2205 is CRITICAL: 7.988e+04 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [10:55:56] (normal) [11:18:53] arnaudb: done [11:19:17] thanks! [11:19:57] but I would give priority to T374425 [11:19:58] T374425: db2205 stuck replication/processlist - https://phabricator.wikimedia.org/T374425 [11:20:05] i'm on it already! [11:20:19] nobody is going to give you a bad time if you don't finish regular maintence [11:20:35] specially if you have a good excuse like that ticket! [11:20:50] so please don't stress about rushing it [12:17:48] RECOVERY - MariaDB sustained replica lag on s3 on db2205 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2205&var-port=9104 [12:55:30] Amir1: https://gerrit.wikimedia.org/r/c/operations/software/+/1072168 seems a duplication of the work done in T371351 or am I missing something else? We have two cookbooks for that since end of july that were reviewed and tested and that will be used for the switchover. (cc jynus as you commented on the CR). [12:55:31] T371351: Automate the pre/post switchover tasks related to databases - https://phabricator.wikimedia.org/T371351 [12:59:46] I thanked Amir as I asked him to publish it for public review/tracking, rather than being on a paste [13:00:21] volans: That is intentional. For this dc switchover, I'm planning to use the most basic version I wrote because it I couldn't well understand the written cookbook, this is an extremely sensitive operation (if it messes up, it's going to cause data corruption and or split brain which is really hard to clean up) and specially since Manuel won't be around to help in case things go wrong. So I wrote something that basically [13:00:21] emulates bash [13:00:37] for later switchovers, we can use the cookbook [13:01:31] I totally and strongly disagree [13:01:36] for so many reasons [13:05:01] I understand but I am responsible for setting up circular replication in production before the switchover and I can't run something I'm not comfortable with in our production, specially for something this sensitive. [13:09:04] the cookbooks have been ready for review and testing since August 1st and my understanding was that they were tested by Arnaud and Manuel already [13:09:38] I think we should discuss this more in depth, I though there was more agreement on this topic than I thought [13:10:36] no concern was voiced in the task or in any meeting I've been in, this is a total surprise for me [13:10:43] it was never tested in actual production section (except test-s4) that worries me a lot. We have specific GTID stuff in production that are different than standard setups [13:11:25] and test-s4 was tested the last day before Manuel's leave [13:11:25] I suggest let's discuss this next monday [13:13:32] sounds good [13:15:31] going for lunch, clearly there has been some misscommunication here, please keep it cool (as it has been so far) 0:-) [13:17:58] arnaudb: one question (no rush on getting an answer)- did you see the semisync issue happening before the script upgrade? Did you used it successfully for some time (trying to discard that as a possible cause) [13:19:50] I'm not sure which script you're referring to jynus [13:21:10] hey peeps, last run of mediawiki_job_purge_parsercache_pc5 failed [13:21:12] Sep 11 01:00:00 mwmaint1002 mediawiki_job_purge_parsercache_pc5[1649]: InvalidArgumentException from line 1388 of /srv/mediawiki/php-1.43.0-wmf.21/includes/objectcache/SqlBagOStuff.php: Unknown server tag: pc5 [13:21:14] the switchover? [13:21:26] yes [13:21:29] Is it safe to restart, or should I let it be and it'll run next time? [13:21:31] claime: that's partially me [13:21:49] I'm bringing it online today [13:21:50] claime: being setup [13:21:53] aaah [13:21:55] ok [13:22:01] but claime it shouldn't have even tried to run it [13:22:12] how it started running it before I got it online [13:22:15] arnaudb: maybe it started to happen after the last patch or something? [13:22:36] I couldn't well understand the written cookbook, this is an extremely sensitive operation (if it messes up, it's going to cause data corruption and or split brain which is really hard to clean up) and specially since Manuel won't be around to help in case things go wrong [13:22:43] wrong paste [13:22:46] I am trying to figure out reasons why it happened to avoid it in the future [13:22:48] https://gerrit.wikimedia.org/r/c/operations/puppet/+/972382/2/modules/profile/manifests/mediawiki/maintenance/parsercachepurging.pp [13:23:00] claime: We need to add them explicitly here, I haven't done that :D [13:23:09] (that's pc4 for last year) [13:23:45] jynus: regarding switchover, we have done a lot of switchovers since the upgrade, that'd be weird if it breaks only for that one [13:24:00] thanks, that is what I was asking [13:24:02] still might be possible but I don't know [13:24:09] Amir1: It got added (I suppose by mistake) in Id4bebbcfa86d4a2539ff58d8868d4a98be7b17c4 [13:24:25] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071750/6/modules/profile/manifests/mediawiki/maintenance/parsercachepurging.pp [13:24:29] claime: ah, I missed it, sorry [13:24:31] Amir1: sure, but if it certainly is with the info you gave me less likely [13:24:42] okay then, I'm putting it online today, it shouldn't cause issues [13:24:57] ah yep, mybad ↑ [13:25:01] sorry! [13:25:03] Amir1: cool, just wanted to make sure [13:25:21] re: claime I'd suggest acking the alert for now with expiration [13:25:35] yep that's what I was gonna do [13:25:42] there are some round dependencies there that sometimes are hard to solve [13:25:58] you need things properly setup to work, and you need things to work to properly setup [13:26:08] *just stateful things* [13:26:14] I need to create the tables first, which I have done last year https://phabricator.wikimedia.org/T350367#9309645 but I don't remember how [13:26:18] * Amir1 kicks his old self [13:26:29] (for not documenting better) [13:26:40] I think there is a maintenance mw script [13:26:46] which one? idk [13:26:52] acknowledged for one day [13:28:59] arnaudb: not your fault, we just missed downtimeing it while it was being setup [13:40:25] aha, I wrote a script for it [13:57:20] pc5 is live in mwdebug1002 now [14:06:13] it works fine [14:12:20] now the fun part of deploying this everywhere [14:25:57] urandom: thanos-fe2004 is on our list of hosts to move switch later, are you able to depool it? [14:25:59] https://phabricator.wikimedia.org/T373101 [14:56:06] topranks: yes, I'll get it [14:56:38] topranks: the window starts in ~1 hours though yes? [14:57:19] yep we're an hour away, and flexible if you need longer just let me know [14:58:02] No no, it's fine. Just wanted to make sure I didn't get the time wrong 😀 [14:58:39] about to pool pc5 [15:00:08] pooled [15:01:14] no explosion [15:05:36] \o/ [15:05:43] I love no explosions [15:06:01] appservers latencies are up but that's intentional: https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&refresh=1m [15:06:09] *expected [15:06:10] its a good indication for movie quality, also for reality [15:06:19] its due to cache warming up? [15:06:31] yup, entries are being displaced [15:23:31] keep disk space monitored, it shouldn't happen, but it wouldn't be impossible that there is an increase due to stale entries (it happened in the past) [15:23:45] *increase in disk usage [15:29:50] good idea jynus: T374551 [15:29:51] T374551: mariadb - monitoring - predict linear on disk/ram usage - https://phabricator.wikimedia.org/T374551 [15:30:26] arnaudb: we triend in the past, but sadly that is very error prone [15:30:53] I would not rely only on this, but it's helped me [15:30:59] for example, when loading data up to e.g. 60% in 1h, that will predict that you will run out of disk space in 2 hours [15:31:08] better have it and not need it than not have it and need it :p [15:31:24] well, we have stuff [15:31:32] we'll have even more :D [15:31:38] thankfully, we did an expansion last year (pc4) which was more disruptive by nature (more keys being displaced) and that didn't cause issues [15:31:47] * arnaudb feels like a tool hoarder [15:31:54] arnaudb: actually that is not necesarilly true [15:32:15] you never know but given that last year it was fine, I'm hopeful [15:32:25] more != better, I'd prefer to work on signal-to-noise ratio rathern than imprerfect signals [15:32:46] that doesn't mean there are no gaps, but we should be careful about them [15:33:02] for example, I would prefer to have that in a dashboard, not a pinging alert [15:33:08] and I belive we already have it [15:33:51] see: https://grafana.wikimedia.org/d/000000106/parser-cache?orgId=1 [15:34:35] Amir1: yeah, that is why I said it was corrected when it was resharded years ago [15:34:51] but for some time it had a lot of issues, I belive Timo helped fixing many of those [15:35:07] and you too I want to remember? [15:35:24] we did some changes and I will do a lot more next Q [15:35:33] T373037 [15:35:34] T373037: Make ParserCache more like a ring - https://phabricator.wikimedia.org/T373037 [15:35:51] in the past I think purges happened only at night [15:36:08] and at some point they started happening continously [15:36:16] for example, we removed 30% of entries by merging mobile and desktop PC entries [15:36:48] if I were me, I would rebuild it on top of something that is not mysql [15:37:06] but you know I am not going to complain about incremental upgrades :-) [15:37:48] I have gone back and forth between mysql and non-mysql and so far has landed on incremental improvements, eventually it should move to another non-mysql solution but future-me problem [15:38:15] arnaudb: just to be clear, I wasn't saying no, just giving context that we tried it in the past and it wasn't easy, so will require some extra work [15:38:45] biggest thing is that right now parsoid for read is being rolled out and I want to make sure they can have "both" entries for at least group of wikis [15:39:04] yeah, and if possible something less hand crafted than memcache + mysql [15:39:22] something that would handle most of that complexity on a different codebase we don't maintain [15:40:11] oh 100% on the same page jynus that makes sense to me, I've been dealing with such rules in the past and the saved me a few times. I'll keep in mind to avoid piling on noise (we have several angles already in the pipes to further improve our current situation) [15:40:44] Amir1: and something that values less C and more AP, rather than mysql regular model of CP over P, that core data requires [15:40:55] yup [15:41:40] arnaudb: there is still a lot of need for more automated capacity planning [15:42:09] this is not really automated capacity planning that i'm aiming for at first iteration but more like a stick to bump on the wall before our collective nose does :D [15:42:10] I hate when I have to go graph by graph on every backup host to see how many servers we need to by in X years [15:42:49] I think that would be a more useful use case for prodictions (long term) [15:42:53] *predictions [15:43:03] yeah but I never managed to get that far in monitoring quality [15:43:14] this would require some datadog like intelligence behind metrics [15:43:42] the capacity planning or the disk full? [15:44:00] long term capacity planning I mean, outside of browsing graphs [15:44:41] not really, for what I need, we could do it a few graphana graphs + prometheus formulas [15:44:56] oh nice [15:45:00] what we need is better organization (and zarcillo doesn't count) [15:45:21] zarciwho? :p [15:58:39] btw, did we investigate why db1166 failed today? [15:59:00] index thingy [15:59:22] i've updated the tracking gsheet [16:01:08] ah, thanks [16:27:39] urandom: all done if you want to repool that host and check the be's [16:27:39] thanks [16:27:58] arnaudb: same goes for the db hosts [16:28:07] topranks: perfect; thanks! [16:29:32] neat, thanks topranks!