[00:34:54] 10DBA, 10Operations: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) data consistency checks have finished for the main tables and it is all fine. This host can get repooled. [00:41:07] 10DBA, 10Operations, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) 05Open→03Resolved This host has been repooled [00:52:24] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Slow query ApiQueryRevisions on enwiki - https://phabricator.wikimedia.org/T224017 (10Marostegui) I ha... [10:15:52] 10DBA, 10Performance: A Query takes suddenly really much too long – something corrupt? - https://phabricator.wikimedia.org/T224656 (10Aklapper) [11:16:20] 10DBA: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10jcrespo) @Bstorm Did your maintenance finish? Can we continue these tasks? [11:17:22] 10DBA: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) Yes, we talked the other day and maintenance is done for now. She might need to do some more next Monday [11:20:06] 10DBA: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10jcrespo) [12:04:46] 10DBA: Querying the first edit of a user takes longer than usual - https://phabricator.wikimedia.org/T224663 (10Ireas) [12:44:48] marostegui or jynus: db1107 has the DIMM issue that needs to be swapped. I want to do this today around 1700-1800UTC [12:45:17] cmjohnson1: you need to coordinate with elukey, he is the service owner and needs to stop all the services there [12:45:24] oh..okay [12:45:24] thanks [12:45:40] cmjohnson1: but he is on an offsite, so I think you might want to postpone it for after our SRE summit [12:46:06] okay I wanted to use it for my on-site interview today....I will find something else [12:48:52] cmjohnson1: I might be able to give you something memory related [12:50:14] cmjohnson1: this host recovered itself: https://phabricator.wikimedia.org/T221502 not sure if it would be useful [12:55:44] that will work [12:55:54] cmjohnson1: let me depool it then [12:55:58] same time frame [12:56:13] cmjohnson1: I will depool it, downtime it and left mysql down, so you can act on it anytime [12:56:22] great! Thanks [12:56:44] cmjohnson1: of potential interest it may be T216240, but as a problem, there is no good actionables there [12:56:44] T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 [12:58:26] that would be something that I will need to do [12:58:43] I am blocking it [12:58:48] becauese I don't think it helps [12:59:49] I think is a hw problem that the bios update doesn't really fix [13:00:31] so very low priority [13:02:36] thumbor1004 is the one I see with active issues, but you should ask filippo or someone else for that one [13:02:55] i see thumbor [13:05:11] cmjohnson1: db1099 is downtimed, and mysql is stopped, you can do anything you like with it, just power it back on when done and update the task letting me know, and I will take it from there :) [13:05:32] okay..thx! [13:06:20] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1099 memory issues - https://phabricator.wikimedia.org/T221502 (10Marostegui) 05Resolved→03Open re-opening as this is going to be worked out. MySQL is stopped on s1 and s8, host downtimed and OS upgraded. It can be taken by @Cmjohnson anytime. Ple... [13:07:16] marostegui: down for at least 24 hours? [13:07:22] (downtime) [13:07:40] jynus: 2 days [13:07:46] cmjohnson1: this might also be a _good_ one too https://phabricator.wikimedia.org/T222731 [13:07:51] marostegui: thanks [13:09:02] cmjohnson1 asks for work, now he regrets doing that after receiving a pile of it! :-D [13:09:10] marostegui: thanks [13:09:11] :) [13:09:22] heh....i knew db1133 would come up [13:09:30] that's on my list...i have to update f/w first [13:09:36] haha [13:10:13] cmjohnson1: also : https://phabricator.wikimedia.org/T213422 [13:10:18] that should be an easy one :) [13:10:56] that is easy...can i do that anytime? [13:11:14] cmjohnson1: it needs depooling too [13:11:33] which I can do now if you want [13:11:40] regardless if I use it or not today..it needs to be done so yes please depool [13:12:09] Yeah, but let's coordinate, as it needs mysql down and everything, so maybe not for today and maybe for a day we are sure it can be done? [13:12:43] I will do it today [13:13:00] ok, let me depool and stop mysql then [13:13:12] I was working on the depooling already [13:13:17] ah [13:13:18] cool [13:13:22] I will leave it to you :) [13:13:27] As I need to get ready :) [13:44:52] jynus: should we upgrade the whole s4 to 10.1.39 so we can promote a 10.1.39 master? [13:45:16] Else, we can remain on 10.1.38 [13:45:39] candidate master is on 10.1.38 [13:46:13] what do you think? [13:48:03] I prefer to upgrade [13:48:13] sounds good! [14:00:54] cmjohnson1: just to be 100% clear, we DON'T have green light yet for es1019, it is blocked by zeljkof [14:01:19] I will ping you when/if I get permission to go on [14:26:45] ok [15:42:37] 10DBA, 10Operations, 10ops-eqiad: db1099 memory issues - https://phabricator.wikimedia.org/T221502 (10Cmjohnson) Swapped DIMM A5 with DIMM B5 and cleared the racadm log. [15:42:46] cmjohnson1: finally es1019 is down and ready [15:43:12] sorry for the delay, not dependen on me [15:43:51] cmjohnson1: thanks for the memory swap [15:44:13] 10DBA, 10Operations, 10ops-eqiad: db1099 memory issues - https://phabricator.wikimedia.org/T221502 (10Marostegui) Thanks - I will take it from here [15:44:14] down until moday just in case [15:46:49] 10DBA, 10Operations, 10ops-eqiad: db1099 memory issues - https://phabricator.wikimedia.org/T221502 (10Marostegui) a:05Cmjohnson→03Marostegui [15:47:13] 10DBA, 10Operations, 10ops-eqiad: db1099 memory issues - https://phabricator.wikimedia.org/T221502 (10Marostegui) mysql started and replication catching up [15:50:48] marostegui: I am going to start handovering to you [15:51:13] I am unsure what to do with labsdb if on monday there is more work to be done [15:51:29] leave me any thoughts you have about that if you want/can [15:57:45] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Slow query ApiQueryRevisions on enwiki - https://phabricator.wikimedia.org/T224017 (10Anomie) >>! In T... [15:58:12] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Slow query ApiQueryRevisions on enwiki - https://phabricator.wikimedia.org/T224017 (10Marostegui) >>!... [16:02:57] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Slow query ApiQueryRevisions on enwiki - https://phabricator.wikimedia.org/T224017 (10Marostegui) >>!... [16:03:38] 10DBA, 10Wikimedia-Site-requests: Global rename of Fiona B. → Fiona*: supervision needed - https://phabricator.wikimedia.org/T224348 (10Anomie) Waiting for the actor migration to get to write-new should indeed reduce the amount of work needed to perform this rename. Current plan is for group 2 (which includes... [16:20:02] 10DBA, 10Wikimedia-Site-requests: Global rename of Fiona B. → Fiona*: supervision needed - https://phabricator.wikimedia.org/T224348 (10jcrespo) @Anomie Regarding specifically quick renames- is is something that will also need to change after migration or is it already bound to the newer actor codebase? [16:54:32] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for vnwikimedia - https://phabricator.wikimedia.org/T207095 (10Urbanecm) 05Stalled→03Invalid Wiki won't be created.