[06:09:15] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10Marostegui) Done! ` | GRANT SELECT, INSERT, CREATE, DROP ON `mwaddlink`.* TO `adminlinkrecommendation`@`10.64.0.135` | 2 rows in set (0.00... [06:09:52] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10Marostegui) @kostajh please test it and if it works fine - feel free to close this task! [06:13:00] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) >>! In T258361#6829557, @Marostegui wrote: >>>! In T258361#6822070, @jcrespo wrote: >> I am taking db1163 to, at least temporarily,... [06:43:52] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1093.eqiad.wmnet - https://phabricator.wikimedia.org/T273955 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db1093.eqiad.wmnet` - db1093.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [06:44:29] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1093.eqiad.wmnet - https://phabricator.wikimedia.org/T273955 (10Marostegui) This is ready for DC-Ops! [06:44:31] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1093.eqiad.wmnet - https://phabricator.wikimedia.org/T273955 (10Marostegui) a:05Marostegui→03wiki_willy [06:45:07] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [07:54:33] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10R4356th) This was [[ https://meta.wikimedia.org/wiki/Tech/News/2021/07 | announced ]] in Tech News yesterday. [07:58:56] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10Marostegui) Excellent - thanks [08:08:52] elukey: I would like to stop and reboot dbstore* hosts for kernel upgrade, let me know when I can proceed [08:14:05] marostegui: please go ahead :) I can do it if you want! [08:14:41] elukey: ah, if you take care of it and then update https://phabricator.wikimedia.org/T273281 that'd be awesome! [08:14:58] marostegui: sure, ok if I do it later on during the day? [08:15:04] elukey: absolutely! [08:15:08] <3 [08:15:12] :** [08:18:31] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10kostajh) >>! In T267214#6832151, @Marostegui wrote: > @kostajh please test it and if it works fine - feel free to close this task! That works, but I missed one more. Could... [08:22:26] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10Marostegui) Done: ` root@db1107.eqiad.wmnet[(none)]> pager grep DELETE ; show grants for `adminlinkrecommendation`@`10.64.0.135`; show grants for `adminlinkrecommendation`... [09:18:39] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10kostajh) 05Open→03Resolved >>! In T267214#6832283, @Marostegui wrote: > Done: > ` > root@db1107.eqiad.wmnet[(none)]> pager grep DELETE ; show grants for `adminlinkreco... [09:21:43] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1172 is now replicating on s8, will start pooling tomorrow [09:32:01] 10DBA, 10Data-Persistence-Backup, 10Data-Persistence: Drop unused database "bacula" from m1 - https://phabricator.wikimedia.org/T274809 (10jcrespo) bacula db was already backed up and sent to the archive pool for retention for a few years in case it is needed (it will not be). [09:51:14] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) I've documented the new architecture to support ES backups, with the additional pools: https://wikitech.wikimedia.org/wiki/Bacula#Retention Once the l... [09:59:51] as I mentioned on the -sre channel, I have documented the new backup pools, hopefully it is understandable: https://wikitech.wikimedia.org/wiki/Bacula#Configured_Pools [10:55:29] es backups at es1022 are about to finish, please wait a few minutes if you want to restart that [10:55:42] only doing codfw [10:55:47] ok [12:00:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [12:08:10] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [12:23:40] marostegui, jynus: FYI, there's an WDQS job running on cumin1001, will try to get an ETA how long it takes and reschedule to a different day [12:25:43] moritzm: sounds good, thanks for the heads up [12:26:16] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1154 [] db1150 [] db1145 [] db1144 [] db1130 [] db1124 []... [13:36:33] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [13:39:12] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [13:39:55] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [13:41:40] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [13:43:15] 10DBA, 10Upstream: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) [13:43:18] 10DBA: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) 05Open→03Declined I am going to close this as we'll not work on individual pieces for tendril, but rather we'll work to totally remove it and replace it with orchestrator and something else. [13:43:57] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for trwikivoyage - https://phabricator.wikimedia.org/T271261 (10Marostegui) @Bstorm @nskaggs I have been pinged about the views for this - do you have some estimation on when they'll be created? [13:54:45] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) s2 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [x] db1171 [x] db1170 [] db1162 [] db1155 [x] db1146 [] db112... [13:57:35] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [14:59:27] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for trwikivoyage - https://phabricator.wikimedia.org/T271261 (10Bstorm) I'll try to make sure it gets done this week. [14:59:30] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for trwikivoyage - https://phabricator.wikimedia.org/T271261 (10Bstorm) I'm surprised this is here! I thought I'd cleared all these out. Sorry. [15:08:35] 10Data-Persistence-Backup, 10Epic, 10Patch-For-Review: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562 (10jcrespo) [15:08:57] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) 05Open→03Resolved This is now done, we have full-covered, regularly-scheduled ES cluster backups on bacula. [15:11:08] https://twitter.com/jynus/status/1361694563136602113 [15:21:18] 😅 [15:53:45] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10Marostegui) <3 nice work [17:37:21] marostegui: I failed to deliver in rebooting the dbstores, will do it tomorrow I promise! [17:45:45] elukey: no problem!! [22:23:35] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for trwikivoyage - https://phabricator.wikimedia.org/T271261 (10Bstorm) a:03Bstorm [22:47:46] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for trwikivoyage - https://phabricator.wikimedia.org/T271261 (10Bstorm) 05Open→03Resolved `lang=mysql MariaDB [trwikivoyage_p]> select * from page limit 2; +---------+----------------+-----------------+---------------...