[05:08:55] 10DBA, 10Gerrit, 10Operations, 10Phabricator: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4274454 (10Marostegui) Not sure if the above actions by @mmodell should have shown any changes on the write patterns, but so far, they remain the same https://grafana.wikimedi... [05:16:21] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pswikivoyage - https://phabricator.wikimedia.org/T196359#4274460 (10Marostegui) @Bstorm in order not to block this on the issues we are having with the grant, I have manually run the database and grant creation on all the hosts. Pl... [05:18:02] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4274461 (10Marostegui) @Bstorm in order not to block this on the issues we are having with the grant, I have manually run the database and grant creation on all the hosts. P... [05:35:14] 10DBA, 10Cloud-Services: Prepare and check storage layer for bn.wikivoyage - https://phabricator.wikimedia.org/T196358#4274462 (10Marostegui) 05Resolved>03Open @Bstorm as reported by @Urbanecm , this still doesn't work after a few hours. ``` marostegui@tools-bastion-03:~$ sql -v --cluster analytics bnwik... [07:05:30] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#4274549 (10jcrespo) > @elukey: Just wanted to confirm that the Analytics team will go forward with the replacement of dbstore1002 with a multi instance set up during the upcoming fiscal... [07:30:45] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for sahwikiquote - https://phabricator.wikimedia.org/T196362#4274578 (10Marostegui) @Bstorm in order not to block this on the issues we are having with the grant, I have manually run the database and grant creation on all the hosts. Pl... [07:32:14] 10DBA, 10Cloud-Services, 10Patch-For-Review, 10User-Urbanecm: Prepare and check storage layer for idwikimedia - https://phabricator.wikimedia.org/T193187#4274581 (10Marostegui) @Bstorm in order not to block this on the issues we are having with the grant, I have manually run the database and grant creation... [07:58:28] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4274597 (10Marostegui) [09:13:20] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10Availability, 10Performance-Team (Radar): wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged - https://phabricator.wikimedia.org/T95799#4274768 (10aaron) >>! In T95799#4274424, @tstarling wrote: > I don'... [09:34:22] marostegui: are you running any maintenance script right now? [09:34:43] I am running an alter from neodymium [09:35:01] this is because I want to merge https://gerrit.wikimedia.org/r/439860 [09:35:19] can you wait maybe 1h or so so it is finished? [09:35:23] with a possiblity that the script messes up the password, want to coordinate with you to avoid issues [09:35:23] just in case? [09:35:41] well, that will only have problems when it is used to connect [09:35:59] to read .my.cnf [09:36:08] yeah, but I am running two in a row :) [09:36:11] I guess I can run it on sarin firt [09:36:28] ah yes, that would also be fine with me! [09:41:15] it worked fine,but ping me if you are going to use some .py script [09:41:32] it may need to be updated to the latest verstion to work on labsdb hosts [09:42:00] nope, no .py scripts on my agenda for today :) [09:42:17] so the main thing is [09:42:23] -[labsdb] [09:42:34] +[clientlabsdb] [09:43:07] this is because mysql client requires extra groups to be sufixes [09:45:04] sure [09:45:11] My alter I think will be finished in around 1h or so [09:45:16] (the second one) [09:45:51] oh, no issue now, I merged already [09:46:11] :) [10:07:58] I have started reviewing the steps, but I have marker on the etherpad on line 40 a couple of things to check [10:08:22] great [10:11:24] I guess all those things we'll need to change once db1066 is master [10:11:38] I wanted to generate a list now [10:11:42] to be able to compare [10:11:42] sure [10:11:47] that sounds good [10:12:04] I am seeing that we do have exceptions in puppet, for instance the expire log days, if it is master or slave, it is different [10:12:15] so as soon as we merge and run puppet it will change on config and we have to change it live too [10:21:17] actually, I would change now [10:21:25] it will not hurt [10:21:31] true [10:21:39] I will do it [10:21:45] ah, thanks! [10:22:05] give a quick check to https://gerrit.wikimedia.org/r/439871 instead [10:22:19] doing it! [10:22:51] not a full review, it is not ready [10:23:08] but just getting an idea of what I worked last day [10:24:27] yeah :) [10:27:02] in fact, it should be 30 everywhere [10:27:19] but I had to reduced it because we were running out of space [10:27:26] with 1TB hosts [10:27:36] we could reevaluate it, but now now [10:35:21] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1056 - https://phabricator.wikimedia.org/T193736#4275004 (10Marostegui) [10:36:48] 10DBA: Decommission db1051 - https://phabricator.wikimedia.org/T195484#4275018 (10Marostegui) [10:37:54] 10DBA, 10Patch-For-Review: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4275020 (10Marostegui) [10:38:28] 10DBA, 10Patch-For-Review: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4204024 (10Marostegui) [11:59:56] 10DBA, 10Gerrit, 10Operations, 10Phabricator: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4275191 (10Paladox) phabricator is going to parse the existing refs/changes/*/*/meta commits (no new ones will be added to the queue so this will eventually go down). According... [12:07:26] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4275215 (10Marostegui) [12:07:35] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4275216 (10Marostegui) [12:07:51] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4275217 (10Marostegui) [12:08:01] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4275218 (10Marostegui) [12:15:19] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4275245 (10Marostegui) s1 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1001 [] dbstore1... [12:15:23] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4275246 (10Marostegui) s1 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1001 [] dbstore1002 [] db1124... [12:15:27] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4275247 (10Marostegui) s1 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [... [12:15:29] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4275248 (10Marostegui) s1 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1001 [] dbstore1002 [] db1124 [] db1080... [12:16:17] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4275249 (10Marostegui) [12:16:33] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4275250 (10Marostegui) [12:16:55] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4275251 (10Marostegui) [12:35:41] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4275319 (10Marostegui) [14:49:53] 10DBA, 10Cloud-Services: Prepare and check storage layer for bn.wikivoyage - https://phabricator.wikimedia.org/T196358#4275744 (10Bstorm) Doh! Yes, there is yet another script (for DNS) that doesn't automatically run. It's running now (and specifically creating the aliases). [14:53:53] 10DBA, 10Cloud-Services, 10Patch-For-Review, 10User-Urbanecm: Prepare and check storage layer for idwikimedia - https://phabricator.wikimedia.org/T193187#4275766 (10Bstorm) Creation is finished. Waiting on dns scripts before close. [14:57:10] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for sahwikiquote - https://phabricator.wikimedia.org/T196362#4275774 (10Bstorm) Done with scripts, when DNS creation is complete, will close. [14:57:26] 10DBA, 10Cloud-Services: Prepare and check storage layer for bn.wikivoyage - https://phabricator.wikimedia.org/T196358#4275775 (10Urbanecm) 05Open>03Resolved Working, thanks. ``` urbanecm@tools-bastion-02 ~ $ sql --cluster=analytics bnwikivoyage Reading table information for completion of table and colu... [14:58:48] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for sahwikiquote - https://phabricator.wikimedia.org/T196362#4275783 (10Urbanecm) Thank you, seems it is working. ``` urbanecm@tools-bastion-02 ~ $ sql --cluster=analytics sahwikiquote Reading table information for completion of tab... [15:00:09] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pswikivoyage - https://phabricator.wikimedia.org/T196359#4275787 (10Urbanecm) Seems it is working, thanks. ``` urbanecm@tools-bastion-02 ~ $ sql --cluster=analytics pswikivoyage Reading table information for completion of table... [15:00:47] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pswikivoyage - https://phabricator.wikimedia.org/T196359#4275793 (10Urbanecm) 05Open>03Resolved See above. [15:01:43] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4275806 (10Urbanecm) 05Open>03Resolved It is working, thank you. ``` urbanecm@tools-bastion-02 ~ $ sql --cluster=analytics pmswikisource Welcome to the MariaDB monitor... [15:02:26] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for sahwikiquote - https://phabricator.wikimedia.org/T196362#4275822 (10Urbanecm) 05Open>03Resolved See above. [15:02:33] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4275825 (10Bstorm) 05Resolved>03Open Not //quite// done with this one. A couple more steps. [15:04:02] 10DBA, 10Cloud-Services, 10Patch-For-Review, 10User-Urbanecm: Prepare and check storage layer for idwikimedia - https://phabricator.wikimedia.org/T193187#4275849 (10Urbanecm) 05Open>03Resolved Seems it is working. ``` urbanecm@tools-bastion-02 ~ $ sql --cluster=analytics idwikimedia Reading table in... [15:04:04] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4275852 (10Bstorm) 05Open>03Resolved Ok, now this one is done. There were still a couple scripts I hadn't finished on all servers for this, and I didn't want to forget... [15:05:30] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4275859 (10Urbanecm) @Bstorm Is there a way how to access wiki replicas // without //having all the steps completed? [15:55:25] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008#4276024 (10Bstorm) Yup! The steps must be completed on all three replica servers, indexes must be created, which doesn't affect access, and analytics vs. web access depend... [17:27:26] 10DBA, 10Operations, 10ops-eqiad: Bad disk on db1065 - https://phabricator.wikimedia.org/T196806#4276512 (10Marostegui) 05Open>03Resolved The new disk worked fine, thanks!! ``` root@db1065:~# megacli -LDPDInfo -aAll Adapter #0 Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name... [21:03:01] * Hauskatze tags with jynus as {{copyvio}} again :P [22:19:27] 10DBA, 10Gerrit, 10Operations, 10Phabricator: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277313 (10mmodell) @marostegui: I canceled some of the queued jobs which should have helped somewhat. The only thing I know to do beyond this is to stop replicating from gerrit. [23:15:03] 10DBA, 10Gerrit, 10Operations, 10Phabricator: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277521 (10mmodell) I'm deleting queued jobs in batches of 100,000. I've also reduced the number of phabricator workers to 5 (from 10) so overall there should be a reduction in...