[05:08:13] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3442588 (10Marostegui) labsdb1009 finished correctly for the existing labs tables ``` categorylinks PRIMARY KEY (`cl_from`,`cl_to`), templateli... [05:08:28] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3442589 (10Marostegui) [05:15:18] 10DBA, 10cloud-services-team: labsdb1009 crashed while doing an alter table on templatelinks - https://phabricator.wikimedia.org/T170657#3442593 (10Marostegui) 05Open>03Resolved a:03Marostegui The alters finished correctly, I have also stopped the server and started it without any issues, so I am going t... [05:18:11] 10DBA: db2019 has performance issues, replace disk or switchover s4 master elsewhere - https://phabricator.wikimedia.org/T170351#3442596 (10Marostegui) This has clearly change its pattern since the bad this was replaced and it is looking better already: https://grafana.wikimedia.org/render/dashboard-solo/db/mysq... [06:37:48] 10DBA, 10Analytics, 10Analytics-EventLogging: dbstore1002 crashed - https://phabricator.wikimedia.org/T170308#3442644 (10Marostegui) 05Resolved>03Open x1 broke with: ``` Could not execute Update_rows_v1 event on table wikishared.echo_unread_wikis; ``` Looks like that table needs to be reimported [06:41:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3442647 (10Marostegui) db1072 is done: ``` root@neodymium:/home/marostegui# for i in `cat s1_tables`; do echo $i; mysql --skip-ssl -hdbstore1001... [06:41:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3442648 (10Marostegui) [07:33:18] 10DBA, 10Analytics, 10Analytics-EventLogging: dbstore1002 crashed - https://phabricator.wikimedia.org/T170308#3442731 (10Marostegui) 05Open>03Resolved I have fixed the issue with the missing record of the table. [08:20:40] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3442810 (10Marostegui) #cloud-services-team I am going to alter the tables on labsdb1001, I think this will take 3-4 days to complete most likely... [08:29:32] hi, what is the status of dbproxy* reboots, are these as complicated like mysql core servers or would they be feasible? (currently going through the list of systems running on 4.4 for the latest round of reboots) [08:30:27] moritzm: We have two set up proxies, so we should be able to reboot at the time, I would wait till Jaime is online, as he set up those, just in case there is some more magic behind the scenes [08:30:38] Like, we have a pair or proxies per set up [08:32:56] great,I'd also like to add these to https://wikitech.wikimedia.org/wiki/Service_restarts [08:36:01] Sure, let's wait for jynus to make sure they can be indeed rebooted one by one [08:37:19] sure, thanks [08:46:44] which ones? [08:47:22] jynus: morning: ˜/moritzm 10:29> hi, what is the status of dbproxy* reboots, are these as complicated like mysql core servers or would they be feasible? (currently going through the list of systems running on 4.4 for the latest round of reboots) [08:47:36] I think we can reboot one of them at the time, as they are paired, but I wanted you to confirm [08:48:00] no, they do not failover automatically yet [08:48:11] one is passive and the other is active [08:48:22] and dns is used right now [08:48:23] aha [08:48:41] which means it requires syncronization with the 20+ owners [08:48:52] because they use connection pooling [08:49:43] hello everybody, I just started the script to purge eventlogging 2014 data on db1047, when it will finish (and everything looks ok) I'll do the same on dbstore1002 [08:50:45] jynus: I am seeing the DNS yeah [09:07:11] I have created the package of 10.1.15 with systemd on jessie [09:07:15] *25 [09:08:05] I would like to upgrade all 10.1 hosts to it and assume we use systemd on all 10.1 servers or things will get complicated [09:09:02] jynus: can you start with labsdb1009? I was about to load s2 there, but haven't started yet, so if you do that one, I can load it there, then you can move to labsdb1010, and I will start the load after you and same with labsdb1011 [09:09:17] I am not in a hurry [09:09:23] ah ok :) [09:09:31] I wanted your opinion [09:09:43] but once I start, I would like to do all relatively quickly [09:09:57] I think only labs hosts use 10.1 [09:10:09] yeah, labs, db1095 and db1102 [09:10:17] yes, labs and those [09:10:24] can you hold it a few days then? [09:10:29] sure [09:10:37] I don't want to block you though [09:10:38] just wanted your feedback [09:10:43] not a block [09:10:49] the package is already created [09:11:05] ok, then let's wait a bit. I am fine by startring to use systemd [09:11:14] I just want to puppetize If package == 10.1 => systemd [09:11:15] and get to play around with it [09:11:17] sure [09:11:20] makes sense :) [09:11:35] so we do not have hosts with it and hosts without it [09:11:58] I do not intend to upgrade hosts to 10.1 in other cases [09:12:02] yeah, and will make things easier in the future 10.1 always systemd [09:12:20] that is what I thought when I created the package [09:12:30] as I had already done the work for stretch [13:55:22] 10DBA, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3443582 (10elukey) Fixed the predump script that wasn't able to backup to `/srv/backup` on bohrium, now everything should be ok.... [14:05:04] 10DBA, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3443667 (10jcrespo) FYI, we uniformized not a long time ago local dumps going to /srv/backups on database hosts, in case you wan... [14:06:05] 10DBA, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3443674 (10elukey) @jcrespo sorry it is `/srv/backups`, missed a 's' in my last post :( [14:24:46] 10DBA, 10Data-Services, 10WMF-Legal, 10Patch-For-Review: Expose ar_content_format and ar_content_model columns of archive table on Labs replicas - https://phabricator.wikimedia.org/T89741#3443770 (10Umherirrender) Adding wmf-legal as suggested on the patch set [14:37:54] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3443849 (10Marostegui) s6 is now available on the new labs servers. I have created the views too. s2 is being imported now [14:37:59] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3443850 (10Marostegui) [14:50:18] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3443883 (10Marostegui) As per T168782#3443867 the first user was created. I can see it in production but correctly filtered on labs. I will wait for a few more to be created... [18:33:50] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10ArchCom-RfC (ArchCom-Approved), and 3 others: Should we bump MediaWiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3445216 (10Reedy) [18:51:42] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10ArchCom-RfC (ArchCom-Approved), and 3 others: Should we bump MediaWiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3445517 (10Reedy) So just #documentation left to sort out... [19:24:31] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10ArchCom-RfC (ArchCom-Approved), and 3 others: Bump MediaWiki's minimum supported MySQL Version to 5.5.8 - https://phabricator.wikimedia.org/T161232#3445690 (10Aklapper)