[05:26:16] 10DBA, 10Data-Services: Prepare and check storage layer for arbcom_ruwiki - https://phabricator.wikimedia.org/T262832 (10Marostegui) p:05Triage→03Medium This is done. One the database is created, please ping us so we can double check it has been correctly sanitized on labsdb hosts. [05:28:16] 10DBA, 10Operations, 10observability: Prometheus/MariaDB counts a 'SELECT ... FOR UPDATE' query as an UPDATE query - https://phabricator.wikimedia.org/T262579 (10Marostegui) I am not sure there's much else to do here. I believe that the most likely explanation on why they aren't on the binlog is T262579#6453... [05:33:33] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) Thank you Papaul. I have started MySQL back in db2125 so replication doesn't get behind too much. [05:40:01] 10DBA: transfer.py fails when copying data between es hosts - https://phabricator.wikimedia.org/T262388 (10Marostegui) The transfer failed again, after 8TB: ` root@cumin2001:~# transfer.py es2017.codfw.wmnet:/srv/sqldata es2027.codfw.wmnet:/srv/transfer 2020-09-14 14:33:26 INFO: About to transfer /srv/sqldata f... [06:15:57] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10Marostegui) I have depooled labsdb1010, I will stop mysql in a couple of hours [06:16:54] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) On going transfers: es2017 -> es2027 es2011 -> es2028 [06:28:42] 10DBA, 10Operations, 10observability: Prometheus/MariaDB counts a 'SELECT ... FOR UPDATE' query as an UPDATE query - https://phabricator.wikimedia.org/T262579 (10jcrespo) 05Open→03Invalid [07:03:16] 10DBA, 10decommission-hardware: decommission es2014.codfw.wmnet - https://phabricator.wikimedia.org/T262889 (10Marostegui) [07:03:37] 10DBA, 10decommission-hardware: decommission es2014.codfw.wmnet - https://phabricator.wikimedia.org/T262889 (10Marostegui) 05Open→03Stalled Not yet ready - let's give es2027 a few more days. [08:14:31] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10Marostegui) labsdb1010 mysql has been stopped @Cmjohnson please take special care of: dbproxy1018 and dbproxy1019, and labsdb1011 as those hosts are serving a [08:49:18] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [08:52:58] jynus: shall we meet here? [08:56:13] better pm so we dont spam the channel [08:57:51] we try to leave this channel free for bots to spam it instead. ;) [09:21:34] Lovely spam, wonderful spam! [09:22:54] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: New Date - Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10Marostegui) >>! In T261455#6458359, @Marostegui wrote: >>>! In T261455#6423007, @Marostegui wrote: >> Please take extra care with db1087, db1100 and d... [09:23:36] jynus kormat marostegui Did you have a chance to look at https://www.mediawiki.org/wiki/GitLab_consultation and see if you have any comments? [09:24:46] @ meeting, will certainly have a look at it soon [09:25:11] sobanski: It is in my to-do list yep, I was planning to do it on friday and here we are XD [09:25:53] sobanski: this might take a while to read though: https://www.mediawiki.org/wiki/Talk:GitLab_consultation :-) [09:26:02] As long as you didn't specify which Friday, you're still good [09:26:07] haha [09:26:12] And yes, I'm going through that part now [12:03:22] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [13:15:10] does anyone know if new labsdbs are going to be called clouddb, maybe? [13:27:17] there's already a clouddb2001-dev.codfw.wmnet, so seems likely [13:31:18] I've uploaded https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/627502 so connections to them don't break [13:34:04] akosiaris: how's otrs progressing, as expected? [13:35:25] jynus: yes [13:35:45] seems to be on schedule. We 'll know for sure tomorrow morning ofc [13:37:26] sobanski: I have no opinion regarding GitLab, I use git a lot but I don't feel strongly about any layer on top of that [13:38:09] All good then :) [13:38:15] my only specific comment, that I transmitted to Releng is that CI would need a bump in terms of less manual operations [13:38:50] but I will adapt to what is decided, I think I am one of the few people apparently that I don't hate gerrit, but maybe I have not seen enough of other systems [13:47:47] jynus: yeah, they are going to be called cloud as far as I remember [13:48:12] see patch I sent so to be prepared for that [13:51:22] will check - thanks [14:03:42] 10Blocked-on-schema-change, 10DBA, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat) [14:07:49] 10Blocked-on-schema-change, 10DBA, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat) s6 codfw progress: [] db2076.codfw.wmnet sanitarium master [] db2087.codfw.wmnet [] db2089.codfw.wmnet [] db2095.codfw.wmnet sanitarium... [14:08:09] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables thanks to the addition of login data - https://phabricator.wikimedia.org/T261999 (10Marostegui) [14:08:27] 10Blocked-on-schema-change, 10DBA, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat) [14:10:08] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables thanks to the addition of login data - https://phabricator.wikimedia.org/T261999 (10Huji) Numbers seem to go down. Maybe we should enable it everywhere to save some space?! 😆 [14:25:29] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables thanks to the addition of login data - https://phabricator.wikimedia.org/T261999 (10Urbanecm) Perhaps we should look at number of edits, and compare if they're going down or up? But yeah, this is confusing... [14:26:17] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables thanks to the addition of login data - https://phabricator.wikimedia.org/T261999 (10Marostegui) This is size on disk, which can be influenced by many things, those sort of variations are pretty normal, I wouldn't be too worry about it. [14:29:11] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), and 2 others: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10Mholloway) [14:41:08] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), and 2 others: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10jcrespo) > Not sure we use TEXT on many places, I recall using blob... [15:50:28] 10DBA, 10Operations, 10netops, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10ayounsi) [15:52:21] labsdb1010 start after PDU maintenance, will leave it catching up before repooling [15:56:23] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: New Date - Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10RobH) [15:56:35] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: New Date - Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10RobH) [16:11:05] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Papaul) @Marostegui please see below . Asking me to do what we already did when we first had the problem which was to upgrade the server Firmw... [16:14:06] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) @wiki_willy can we escalate this to someone else? This seems a bit of a loop over a kinda new server that shows the same HW error... [16:32:42] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [16:32:48] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [16:36:42] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), and 2 others: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10Mholloway) @Marostegui Our plan is to enable either for all Wikipedi... [16:44:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [17:17:40] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: New Date - Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10RobH) [17:23:40] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: New Date - Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10RobH) [17:35:31] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [17:38:47] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [19:22:27] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10wiki_willy) Hi @Marostegui - I can escalate this to our account rep, and see if they can either escalate up further or swap it out with a new... [20:30:52] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Schema-change, 10User-DannyS712: flaggedpage_config.fpc_select is unused - https://phabricator.wikimedia.org/T262978 (10DannyS712)