[04:50:02] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10Marostegui) [04:50:15] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10Marostegui) [05:10:27] 10DBA, 10Operations: Decommission db2055.codfw.wmnet - https://phabricator.wikimedia.org/T233186 (10Marostegui) [05:13:31] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [06:00:08] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) I have captured 30 minutes of read queries on db1105 (rc on enwiki) and replied them into db1089 (non partitio... [06:22:30] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10wiki_willy) a:03Jclark-ctr [06:27:35] 10DBA, 10Analytics, 10Data-Services: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) I have sanitized hiwikisource on sanitarium hosts (db1124:3313 and db2094:3313) and I have checked that the users were sanitized correctly, and my user, which I cr... [06:29:58] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10wiki_willy) [06:48:08] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) >>! In T202367#5502575, @Marostegui wrote: > dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires... [07:37:14] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10MoritzMuehlenhoff) [07:39:21] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Marostegui) This is what m5 has at the moment: ` +------------------------+ | Database | +------------------------+ | designate | | designate_pool_m... [07:41:31] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [08:06:48] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Krenair) It's the labspuppet database, yes. Note that the toolforge project has its own puppetmasters and only they were talking to the central puppetmaster. [08:07:44] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Marostegui) What we normally do is, rename all the tables on the given database and leave it like that for a few days to check that nothing is really using it, and then drop it. [08:08:56] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Krenair) Sounds like a good idea, let's do it. [08:12:57] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Marostegui) a:03Marostegui Done: ` root@db1133.eqiad.wmnet[labspuppet]> show tables; +-------------------------+ | Tables_in_labspuppet | +-------------------------+ | TO... [08:35:40] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [08:36:42] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) We may need some firmware updates, but hw is ready to go as soon as background raid initialization finishes on array2 of backup1001. Hosts installed with buster.... [09:19:13] 10DBA, 10Operations: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10jcrespo) Reminder: Let's check grants too. [12:08:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10Cmjohnson) This server is out of warranty and @robh has created a procurement task. [12:11:47] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: labsdb1009 broken PSU - https://phabricator.wikimedia.org/T233273 (10Marostegui) >>! In T233273#5505877, @Cmjohnson wrote: > This server is out of warranty and @robh has created a procurement task. Indeed Chris - thanks! I caught up with Willy earlier today and... [12:15:27] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) p:05Low→03Normal a:05Marostegui→03None This is ready for #cloud-services-team - the views can be created on labsdb1009-labsdb1012.... [13:15:02] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) Getting an error when trying to create the views. Maybe similar to what we saw in T193187? ` labsdb1009:~$ sudo /usr/local/sbin/maintain-vie... [13:17:10] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Probably, let me check [13:18:10] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Can you try again on labsdb1009? [13:22:53] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) Yep, it's working now. The views have been created on labsdb1009 and labsdb1012. [13:23:27] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Excellent, you can proceed on labsdb1010 and labsdb1011 too. I have created the DB there too. [13:27:19] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506135, @Marostegui wrote: > Excellent, you can proceed on labsdb1010 and labsdb1011 too. I have created the DB there too. A... [13:29:23] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) DNS and all the other operations done too? I still cannot access the database from the tools bastion: ` marostegui@tools-sgebastion-07:~$... [13:44:18] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) OK, the replica DNS entries are setup for hiwikisource now. I went through the full instructions for cloud services https://wikitech.wikimed... [13:46:35] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) db1089 has been serving pretty almost 8h now the `logpager` service handling 50% of its traffic: ` "logpa... [13:53:07] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10jcrespo) In order to give more context, if partitioning was still needed, query timeouts/long running happen almost immedi... [14:00:40] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) >>! In T219374#5506250, @JHedden wrote: > OK, the replica DNS entries are setup for hiwikisource now. > > I went th... [14:21:20] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) @JHedden after your +1, I have merged the change. Can you run puppet and try again to see if that index error is no... [14:23:21] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) For what is worth, I can access the hosts from bastion with `-- cluster analytics` and `--cluster web` and I can se... [14:26:58] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506354, @Marostegui wrote: > @JHedden after your +1, I have merged the change. Can you run puppet and t... [14:30:18] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) So everything done? From my side I can connect fine and query the views. [14:39:01] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506399, @Marostegui wrote: > So everything done? > From my side I can connect fine and query the views.... [14:39:52] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Let's wait for @Bstorm then before closing this as resolved Thanks! [15:54:17] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Anomie) [16:00:26] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Anomie) >>! In T219374#5506301, @Marostegui wrote: > This might be related to {T233135} cc @Anom... [16:08:22] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Yep, it was intentional [16:37:27] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) That meta_p and maintain-indexes thing will make the wiki non-functional for some purpos... [16:41:43] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) To provide context, we did a lot to prevent breakage of the tables during that refactor,... [16:42:48] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Thanks for providing the context Brooke! I will remove the temporary index tomorrow... [16:44:34] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) Looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/538030 fixes it, but I d... [16:45:08] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) Thanks @Marostegui [17:08:23] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) The problem is apparently a lot of settings being moved from InitializeSettings.php to V... [18:05:51] 10DBA, 10Analytics, 10Data-Services, 10cloud-services-team, and 2 others: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) 05Open→03Resolved a:03Bstorm Ok, I've run maintain-meta_p on all 4 replicas. Docs don't need an update because what I cha...