[04:52:40] 10DBA, 10Commons, 10MediaWiki-Database, 10MediaWiki-Special-pages, and 2 others: Special:Log/Fanghong results in fatal exception of type "Wikimedia\Rdbms\DBQueryTimeoutError" - https://phabricator.wikimedia.org/T199790 (10Marostegui) db1103 and db1097 are both using 10.1.31. I will upgrade db1103 to 10.1.3... [05:47:24] so, you lead? what do you want me to help you with? [05:47:36] sure [05:47:51] If you want to do the monitoring and double check the position once I paste it on the etherpad? [05:48:29] ok [05:51:15] you will need to rebase 445371 after merging the other patch [05:51:45] let's move to ops [05:51:59] yeah, I was doing so now :) [06:15:48] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) This was smoothly done. Read only times: Start: 06:01:31 Finish: 06:07:17 [06:36:59] 10DBA: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) p:05Triage>03Normal [06:39:39] 10DBA: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10jcrespo) I thought a bit about how to go over this, and given the importance and history of this host, this would be one proposal, see what you think about it: * Wait 1 week to make sure we are not going to fail back immediately * Archive... [06:40:41] 10DBA: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) >>! In T199861#4432834, @jcrespo wrote: > I thought a bit about how to go over this, and given the importance and history of this host, this would be one proposal, see what you think about it: > > * Wait 1 week to make sure we... [06:41:58] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) [06:42:48] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) Resolving this as it has all be done - including the clean up tasks. [06:43:27] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320 (10Marostegui) [06:43:29] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) 05Open>03Resolved [06:57:39] 10DBA, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Marostegui) a:03Marostegui [07:16:34] 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10jcrespo) What do you think of doing this next wednesday? [07:19:06] 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10Marostegui) Good for me! [08:34:22] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 (10Marostegui) [08:34:48] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 (10Marostegui) 05Open>03Resolved db1052 was failed over today and now got the schema change so this is all done. [09:05:37] 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10jcrespo) https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1797449&oldid=1797439 [09:40:23] 10DBA, 10Commons, 10MediaWiki-Database, 10MediaWiki-Special-pages, and 2 others: Special:Log/Fanghong results in fatal exception of type "Wikimedia\Rdbms\DBQueryTimeoutError" - https://phabricator.wikimedia.org/T199790 (10Marostegui) I will be running this on db1103 to let it with the same schema as the re... [09:41:25] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 (10Marostegui) [09:41:45] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 (10Marostegui) 05Open>03Resolved db1052 was failed over today and now got the schema change so this is all done. [11:02:33] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 (10Marostegui) db1052 was failed over today and now got the schema change so this is all done. [11:03:18] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 (10Marostegui) 05Open>03Resolved [12:05:08] ok to install PHP security updates on dbmonitor*? should only cause a few seconds on non-availability during the restart [12:08:06] yes [12:12:38] doing that now [12:15:06] done, tendril continues to work fine for me [12:17:55] jynus: marostegui Tomorrow I'm planning to deploy a change that will increase the write on change_tag table and I start to backpopulate the new column everywhere. Is it fine? It might take days I will add it to the deployment calendar [12:18:24] it will be basically writes of writes on the table. Any section I should avoid for now? [12:19:37] *increase [12:20:24] what does write to that table? synchronously with rcs or jobqueue or something else? [12:22:39] done, tendril continues to work fine for me [12:23:22] jynus: the maintenance script and also the mediawiki itself when someone makes an edit, etc. [12:24:45] just the usual thing, e.g. start with production testing on testwiki then grow from there, I trust you to do the right thing [12:25:24] did you check with manuel the column is ready everywhere? [12:55:23] Yeah, it should be everywhere, I checked [12:55:27] But probably worth checking again [12:55:29] I will do that [12:57:12] Amir1: ct_tag_id right? [12:57:40] jynus: yeah [12:58:00] marostegui: yes, and also the new table change_tag_def [12:58:21] jynus: We already deployed it on wikiousrce, wikiboyage, wikibooks and frwiki [12:58:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 (10Marostegui) [12:58:41] I will do a "quick" pass to make sure there are no slaves without it [13:05:44] heh, db1118 the mysql 8.0 test host in enwiki was missing the column :) [13:05:47] I will alter it [13:06:54] I think db1118 wasn't part of s1 for a long time [13:07:00] yeah [13:07:11] we may have to scratch it and rebuild it [13:07:39] yeah, probably [13:07:46] I will alter it for now so replication won't break [13:07:51] so I wouldn't spend much time on it [14:31:44] Amir1: you should be good to go, I have checked the column across the hosts and it is there [15:35:37] 10DBA, 10MediaWiki-extensions-Newsletter, 10Patch-For-Review, 10Performance: List of Newsletters should have a column showing the number of issues - https://phabricator.wikimedia.org/T180979 (10Pppery) a:05Pppery>03None [15:36:02] 10DBA, 10MediaWiki-extensions-Newsletter, 10Patch-For-Review, 10Performance: List of Newsletters should have a column showing the number of issues - https://phabricator.wikimedia.org/T180979 (10Pppery) [15:44:17] marostegui: Great, thanks! [16:52:42] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 23): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449 (10dbarratt) [16:54:13] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 23): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449 (10dbarratt) [18:03:07] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1056 - https://phabricator.wikimedia.org/T193736 (10RobH) a:05Cmjohnson>03None [18:38:04] 10DBA, 10MediaWiki-Maintenance-scripts, 10Operations: sql enwik gives a poor error message when db doesn't exist - https://phabricator.wikimedia.org/T199008 (10herron) p:05Triage>03Low [20:28:56] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Extension:JADE scalability concerns due to creating a page per revision - https://phabricator.wikimedia.org/T196547 (10Krinkle)