[03:09:41] 10DBA, 06Community-Tech, 06Stewards-and-global-tools: Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2948316 (10TTO) [07:10:16] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 crashed (RAID controller) - https://phabricator.wikimedia.org/T154031#2948528 (10Marostegui) After the reboot the Cache looks good now ``` Cache Status: OK ``` Going to repool the server for now as it looks stable for the past few weeks. [07:18:05] 10DBA, 06Community-Tech, 06Stewards-and-global-tools, 13Patch-For-Review: Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2948316 (10Marostegui) Hi! Thanks for the detailed summary. Can you please add me as a reviewer to the SQL patch so I can take a look? Once this tic... [07:39:22] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2948559 (10Marostegui) For the record: I have restarted dbstore2001 mysql to manually apply the variables to make innodb the default storage engine as well as increasing its buffer_pool size as I didn't wait... [07:46:17] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 10Revision-Scoring-As-A-Service-Backlog: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2945272 (10Marostegui) Hello, Thanks for the ticket. I have been taking a look and this looks related to... [08:22:31] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2948614 (10Marostegui) The following tables have been compressed across all the wikis: ``` revision templatelinks pagelinks ``` This is the situation with this host now: ``` root@db1044:/srv/sqldata# df -hT /srv/ Filesystem Type Size U... [08:23:02] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2948617 (10Marostegui) 05Open>03Resolved p:05Triage>03High [08:34:36] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 crashed (RAID controller) - https://phabricator.wikimedia.org/T154031#2948654 (10Marostegui) 05Open>03Resolved a:05jcrespo>03Marostegui [09:51:48] 10DBA, 13Patch-For-Review: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2948788 (10Marostegui) db2063: ``` root@neodymium:/home/marostegui/git# mysql -hdb2063.codfw.wmnet enwiktionary -e "show create table templatelinks\G" --skip-ssl *********************... [10:12:08] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2948820 (10Marostegui) I am going to enable gtid_domain_id first on dbstore2001 before doing it on m3 to make sure it is all good. It is the first time we enable it since the tests on a multisou... [11:12:44] 10DBA, 13Patch-For-Review: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2948894 (10Marostegui) db2064: ``` root@neodymium:/home/marostegui/git# mysql -hdb2064.codfw.wmnet enwiktionary -e "show create table templatelinks\G" --skip-ssl *********************... [11:36:46] 10DBA, 13Patch-For-Review: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2948932 (10Marostegui) Only pending the master in codfw. This is a non-online operation, so once I decide do it on the master the slaves will get lagged, since it is codfw it should be... [12:23:14] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2949006 (10Marostegui) I have deployed puppet and manually enabled the gtid_domain_id flag on m3 (phabricator) no problems have been encountered and replication is flowing fine on the slaves and... [13:03:19] 10DBA, 06Labs, 10Tool-Labs: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2949041 (10Marostegui) @yuvipanda let me know when you want to do this Thanks [13:03:52] marostegui: wanna do it now? https://phabricator.wikimedia.org/T151014#2949041 [13:04:00] sure [13:04:17] marostegui: ok, I'm going to watch the account creator thing [13:04:18] want to start with labsdb1001? [13:04:22] ok [13:04:35] sure [13:05:13] let me backup the users [13:05:18] just in case we need to create them again [13:06:30] ok, I am ready to drop labsdbadmin @10.64.37.{6,7} on labsdb1001 whenever you want [13:10:28] labsdb1004 and 1005 also have these users, so I guess we can delete from there too [13:33:59] yuvipanda: any issues on labsdb1001? the users are gone now [13:36:00] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2949078 (10faidon) Ping! Jan 25 is a week away from now, not a lot of time left for an announcement :) [13:37:54] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2949082 (10mark) p:05Normal>03High [13:38:01] marostegui: sorry, was away. Everything is ok [13:38:20] yuvipanda: shall I go ahead with 1003,1004 and 1005? [13:38:35] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2949083 (10yuvipanda) I didn't manage to send out the announcement due to unforseen personal issues. I'll send it out now after checking with jynus. [13:38:37] marostegui: yes! [13:38:46] \o/ [13:38:55] ok, will do now and update+close the ticket [13:39:04] thanks! [13:39:32] marostegui: jynus how long do you think the toolsdb migration will take? [13:41:00] 10DBA, 06Labs, 10Tool-Labs: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2949085 (10Marostegui) This has been executed on the following hosts: ``` labsdb1001 labsdb1003 labsdb1004 labsdb1005 ``` ``` set session sql_log_bin=0; drop user 'labsdbadmin'@'10.64... [13:41:43] yuvipanda: jaime is out today and possibly tomorrow too. Do we have to migrate OS and Maria to 10.0, right? [13:42:29] marostegui: yeah [13:43:51] I am unsure about how Jaime usually does (we never talked about it) it and if there is some things that need extra care, but I guess 1 hour or so? [13:44:18] I would wait for him on this [13:51:07] he might be checking emails, but irc probably not, so maybe you want to send him an email [13:54:20] marostegui: hmm, if I schedule say 4h would that be enough [13:54:38] yuvipanda: without having all the context I would say so [13:57:19] marostegui: ok! including preserving the data, right? [13:57:43] yuvipanda: I assume it is just an OS upgrade + upgrading mariadb [13:57:48] so data will remain untouched [13:57:53] apart from the normal mysql tables upgrade [13:58:01] mysql internal tables I mean [13:58:35] marostegui: ah, I see. this host is a bit... special as in I don't know how puppetized it is, so maybe it's a bit more fucked than that [13:59:07] yuvipanda: could be, that is why I am unsure. however, a normal apt-get dist-upgrade and a normal apt-get update the mariadb package would be expected, but I am not really sure [13:59:12] 4h should be fine I hope [13:59:43] marostegui: ok! we usually also re-image but idk about dbs [13:59:55] ah [14:00:02] let me check the size of the dataset [14:00:21] marostegui: yeah, I don't know if we've ever done a dist-upgrade, and from precise to jessie [14:00:36] oh, precise [14:00:38] yeah [14:00:42] reimage then XD [14:01:20] marostegui: yeah... [14:01:33] the dataset is 1.6T so that is gonna take around 1:30h to copy somewhere else, and then back again [14:13:13] 10DBA: db1026 (s5) needs some compression - https://phabricator.wikimedia.org/T154929#2949148 (10Marostegui) a:03Marostegui [14:49:40] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949174 (10Paladox) @jcrespo hi, apparently you can set mutiple mysqld for example # [mysqld2] # port = 3307 # datadir... [14:59:00] marostegui: ok, so 6h? [14:59:07] I'm sure something will go wrong, that box is... special [14:59:17] yuvipanda: ok, let's go for 6h [14:59:26] we discovered it was unpuppetized under interesting ciscumstantes about 3y ago [14:59:30] marostegui: ok! [14:59:30] oh really? [14:59:42] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949178 (10Marostegui) >>! In T145885#2949174, @Paladox wrote: > @jcrespo hi, apparently you can set mutiple mysqld > > for exam... [15:00:16] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949179 (10Paladox) oh [15:10:19] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949187 (10Paladox) @Marostegui i guess we should do the conversion of db, it will at least stop gerrit making error's. It will j... [17:02:06] 10DBA, 06Analytics-Kanban: Sqoop doesn't run anymore - Seem related to a DB change (analytics store) - https://phabricator.wikimedia.org/T154685#2949425 (10JAllemandou) Removing limitations has solved the issue. [17:11:09] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949434 (10Paladox) @Marostegui hi, i managed to do mysql_multi, it took a while to setup as i have never done it. but in the end... [17:20:59] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949474 (10Marostegui) >>! In T145885#2949434, @Paladox wrote: > @Marostegui hi, i managed to do mysql_multi, it took a while to... [17:23:21] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2949483 (10Paladox) oh, sorry i didn't realise that the firewall / puppet and monitoring would need to be changed. Is there any... [21:45:38] 10DBA, 06Community-Tech, 13Patch-For-Review, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2950570 (10MarcoAurelio)