[05:09:39] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, and 2 others: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) After merging puppet I ran the following on the 4 labsdb hosts to remove the temporary index on the hiwikisource view for ar... [05:23:07] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) After leaving db1089 all night this is the only query reported involving `logging` table that has some differe... [07:52:51] good morning :) [07:52:59] I am available to talk about https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/538045/ whenever you have time [08:02:48] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) So, running the analyze didn't change things on db2102, it kept choosing `log_actor_type_time` However an `alt... [08:03:37] elukey: that looks ok, but maybe a PCC check would be nice, and I guess the first backup should probably be done manually to verify things as well [08:05:16] marostegui: ah yes yes I didn't mean to merge, just to know if the new config was ok [08:05:26] I'll run a pcc now though [08:43:42] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) For what is worth and to give some perspective, the above query has only been reported 13 times in 24h, so it... [08:56:43] elukey: the change itself looks good, but probably needs grants deployments and so on, probably you want to coordinate with Jaime to also generate a manual backup [08:59:05] yep! [08:59:14] I haven't created the dump user on the dbs yes [08:59:16] *yet [09:02:55] 10DBA, 10Operations, 10ops-codfw: db2127 memory issues - https://phabricator.wikimedia.org/T233184 (10Marostegui) No more errors, if it continues clean on Monday I will close this task [09:14:29] Package bacula-sd-mysql is not available [09:14:36] However the following packages replace it: [09:14:42] bacula-sd bacula-bscan [09:15:21] buster? [09:15:26] or maybe bacula-director-mysql ? [09:15:33] puppet needs some ifs [09:15:58] https://packages.debian.org/source/buster/bacula [09:16:11] I guess the director one is [09:16:55] I am going to run puppet on icinga, then disable alerts [09:20:07] trivial changes are ok and expected [09:20:16] as long as they are only those [09:22:20] also "User name was not found" [09:22:38] ah, probably the renamed package create it on preinst [09:23:01] this is nice because I get better understanding [09:26:24] akosiaris: question, did you import the bacula module from somewhere or did you created it from 0? [09:26:42] jynus: from scratch [09:27:00] cool! [09:46:04] oh, wow, it even has tests! [10:05:34] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [10:46:23] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) Almost there: > ` > > > Config error: Cannot open config file "/etc/bacula/bacula-sd.conf": Permission denied > > ` [14:24:11] 10DBA, 10Operations, 10Patch-For-Review: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10MoritzMuehlenhoff) >>! In T233281#5505493, @jcrespo wrote: > Reminder: Let's check grants too. Created https://gerrit.wikimedia.org/r/538270 for this [16:01:57] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Anomie) All that reminds me of T71222#4994266. We might have to add index ignoring to LogPager like we did for ApiQueryLog... [16:37:55] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) Yeah, I believe so too, so +1 to include the IGNORE. I am going to create a page on wikitech to document the... [19:41:54] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review, 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Anomie)