[11:30:00] marostegui: do you have any helpful info regarding rebooting Bacula director and storage daemon instances? [11:30:32] cezmunsta: None :( [11:34:00] :D OK, my plan is to disable the schedules for (weekly, hourly and daily exist) and then stop the director, stop SDs, umount and then issue the reboot cookbook across them all [11:35:21] I don't see when you cross your fingers [11:35:54] They are already crossed, lock will be released afterwards [11:36:00] hahaha [13:06:38] Well, an incremental backup completed, so I am counting that as back in service [13:07:24] 👍 [13:08:59] cezmunsta: thanksss! [13:39:55] darn it, Dell didn't even cable up these 3 nodes the same [14:04:05] something is going on in s2 [14:04:14] general lag there [14:05:24] it is gone now, in eqiad, still processing in codfw [14:06:25] grafana super slow for me, but I want to check the master, I think it was a large transaction/spike of writes [14:08:33] Grafana seems OK for me [14:08:54] yeah must be me :( [14:09:10] ... well actually, that depends on what I look at [14:09:28] mysql-replication-lag is slow [14:09:54] whereas mysql-aggregated is OK [14:10:24] There is a large spike in rows being written to s2 though [14:10:33] right, that is what I thought [14:10:51] Could I get a review of https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/23 (to teach the ring manager about a new-to-swift rack) and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357295 to add 3 new nodes (with annoying device paths), please? [14:11:25] marostegui: it started ~30 minutes ago [14:11:43] Emperor: looking [14:12:34] marostegui: are you logged in to Grafana? [14:12:35] cezmunsta: yeah, huge spike https://grafana.wikimedia.org/d/000000273/mysql?from=2026-10-20T11:44:04.010Z&to=2026-10-20T14:11:52.680Z&timezone=utc&var-job=$__all&var-server=db1162&var-port=9104&refresh=1m&viewPanel=panel-33 [14:12:39] cezmunsta: nope [14:12:52] I am doing so after readnig -sre [14:13:32] Yep, if I look at the anonymous endpoint then it does seem to behave differently [14:15:05] actually, almost all sections seem to be elevated [14:16:50] https://grafana.wikimedia.org/d/000000278/mysql-aggregated?from=now-6h&to=now&timezone=utc&var-site=eqiad&var-group=core&var-shard=$__all&var-role=$__all&viewPanel=panel-19 [14:16:54] this is quite crazy [14:17:00] Amir1: any idea if something has been pushed? [14:17:19] I don't see anything relevant in salt [14:17:21] sal [14:17:42] ah sorry. let me check I was afk for lunch [14:18:02] what is this? https://grafana.wikimedia.org/d/000000278/mysql-aggregated?from=now-6h&to=now&timezone=utc&var-site=eqiad&var-group=core&var-shard=$__all&var-role=$__all&viewPanel=panel-16 [14:18:09] Interestingly, s1 seems to be one of the outliers, along with ms* [14:18:35] the writes are mostly on s2 from what I'm seeing. Right? [14:19:21] Amir1: yeah and s5 had a spike but it is mostly s2 [14:19:36] very likely a popular template got edited [14:20:04] This is also quite weird: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?from=now-6h&to=now&timezone=utc&var-site=eqiad&var-group=core&var-shard=$__all&var-role=$__all&viewPanel=panel-16 [14:20:25] I can take a binlog and do split based functions, I've done that a couple of times I should have the bash command somewhere even [14:20:57] Nothing at 12:00 that can explain that huge increase too [14:21:11] for the ssl one, I have some idea: That only happens if requests to the secondary dc end up needing to connect to a primary [14:21:28] Amir1: right, but why it's not come back down? [14:21:42] which beside the ssl thing, should also trigger logs, so we can check logstash [14:22:12] Did I see something from effie about testing traffic? [14:22:37] 12:03 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896 [14:22:38] that? [14:22:47] let me see [14:22:48] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [14:23:46] * cezmunsta wafts away the smoke from the potential barrel [14:23:49] 11:58 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T440896 [14:23:50] https://logstash.wikimedia.org/goto/b5869da88489887beceb92d13ca466cf [14:23:52] it does match the time [14:23:56] this could be a factor [14:24:32] yeah, that would trigger a lot of cross dc calls [14:24:43] hello world, how can I help [14:25:12] effie: That huge increase in that graph could mean we are doing lots of cross dc queries, can that be the case with your deployment? [14:25:18] we need to take over troy [14:25:25] cdanis: the s2 writes thing: [16:19:37] very likely a popular template got edited [14:25:31] Amir1: my specialty [14:25:41] let me see what I did at ~12 UTC [14:25:47] effie: [14:25:52] 11:58 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T440896 [14:25:55] 12:03 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896 [14:26:05] that looks like me yes [14:26:06] Amir1: that logstash says metawiki, that's s7 right? [14:26:09] effie: ok [14:26:38] yeah and timing doesn't match, has been going on for a while [14:26:44] but should be fixed regardless [14:26:48] I'll file a ticket [14:26:57] I am planning to move mw-web after luca is done restarting [14:27:19] any concerns to postpone? [14:27:36] effie: no, it is fine, just making sure we understand that increase on that graph, but it is all good [14:27:47] alright, thank you folks [14:28:27] I'd say be careful though as a lot of db queries will take a massive latency hit so things will get very slow [14:28:29] Amir1: do you want me to check binlogs in s2 master? [14:28:42] sure thing, I actually haven't finished my lunch yet [14:28:53] I'd be grateful so I can finish it [14:29:01] ok, let me see if I can see somethign interesting there [14:29:14] <3 [14:29:27] interestingly the graphs for s2 master doesn't show any massive increase [14:29:36] but it does show on write disk io [14:29:37] weird [14:32:28] maybe transactions with massive updates in it? [14:32:55] Second highest in sys for io_global_by_file_by_bytes = @@datadir/enwiktionary/templatelinks.ibd [14:33:21] Amir1: top items: [14:33:22] 68237 update /* MediaWiki\JobQueue\Jobs\HTMLCacheUpdateJob::invalidateTitles [14:33:22] 28907 update /* MediaWiki\Extension\DiscussionTools\ThreadItemStore::insertThreadItems [14:33:22] 12488 update /* MediaWiki\Deferred\LinksUpdate\LinksUpdate::updateLinksTimestamp [14:33:35] from 13:45 to 14:20 [14:37:19] that looks like a template edit in enwiktionary [14:37:37] now I want to know how cezmunsta got that data :D [14:38:14] I want to add something that may or may not be related [14:38:33] Amir1: the table lives in sys schema [14:38:37] I have tihis pattern here [14:38:37] https://grafana-rw.wikimedia.org/goto/szc6jq?orgId=default [14:39:17] which means there are 3 (?) keys that being requested periodically, a lot, and I suspect [14:39:30] Amir: yes, in sys and as marostegui said disk I/O and not writes as such, I figured bytes was a good place to look [14:40:12] effie: but I don't think that is related to the s2 writes [14:40:32] i am looking if it is something taht is generated or created over and over again [14:40:49] Amir1: Those top updates ring a bell then? [14:41:00] but I do not see anythig to back this, so sorry for the noise [14:42:26] marostegui: they look like template updates but do you have the db too? That should be easy to get [14:43:04] effie: do you have the keys? I'm kind of curios if we can make it not do that :D [14:43:30] Amir1: not yet, because I need to dump traffic there [14:44:03] let me know and I can try to do something about it from mw side of things [14:46:02] Amir1: let me see [14:46:17] Amir1: they ar causing trouble because they exhaust the network [14:46:31] but lets see, thanks ! [14:47:30] causing trouble is the job of thumbor :( [14:47:46] 65203 update itwiki /* MediaWiki\JobQueue\Jobs\HTMLCacheUpdateJob::invalidateTitles [14:47:46] 8766 update itwiki /* MediaWiki\Extension\DiscussionTools\ThreadItemStore::insertThreadItems [14:47:47] 6837 update enwiktionary /* MediaWiki\Extension\DiscussionTools\ThreadItemStore::insertThreadItems [14:47:47] 4421 update enwiktionary /* MediaWiki\Deferred\LinksUpdate\LinksUpdate::updateLinksTimestamp [14:47:51] Amir1: ^ [14:48:15] Amir1: we sorted that poor sod out [14:51:18] marostegui: https://it.wikipedia.org/w/index.php?title=Modulo:Lingue/Configurazione&curid=4589629&diff=152989146&oldid=152877233 [14:51:23] https://templatecount.toolforge.org/index.php?lang=it&namespace=828&name=Lingue%2FConfigurazione#bottom [14:51:38] transcluded in 2.2M pages [14:51:46] that matches the time yeah [14:52:56] So I guess we've solved the mystery [14:54:27] which means I need to update templateradar to include the italian wikipedia [14:54:34] it never stops bothering me how wasteful the whole "refresh all pages using this template because someone made a tiny update" is [14:55:53] Amir1: and scary