[05:31:39] 10DBA, 10MediaWiki-Database, 10MediaWiki-Special-pages, 10Security, 10Wikimedia-log-errors: Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way. - https://phabricator.wikimedia.org/T191116#4103154 (10Marostegui) There are still some errors: https://logstash.wikimedia.org/goto... [05:52:17] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#4103159 (10Marostegui) [05:52:44] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4103162 (10Marostegui) [05:52:47] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#1737574 (10Marostegui) 05Open>03Resolved I have dropped the table everywhere where... [06:05:42] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 2 others: Investigate optimzing wb_terms - https://phabricator.wikimedia.org/T188279#4103192 (10Marostegui) db2083 is now depooled and with alert notifications disabled. Assuming @Ladsgroup will use the wikiadmin user to... [06:19:13] 10DBA, 10MediaWiki-Platform-Team, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4101112 (10Marostegui) So this is all ready? (pending the merge and finishing with T185128). I am asking to see whether we should add #blocked-on-schema-change alrea... [06:21:01] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 2 others: Investigate optimzing wb_terms - https://phabricator.wikimedia.org/T188279#4103196 (10jcrespo) > We can either change the password, or the user, or both. +1 [06:22:30] Good morning and thanks for the patch! [06:23:29] just to be clear again- the server we are going to give you is in production (so data cannot leave the server) [06:23:53] it is just that you will be free to break it in any way possible [06:26:20] marostegui: it also needs read-write, do I change it? [06:35:22] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4103215 (10jcrespo) Hey, @Marostegui, when deleting tables, can you check views on labs, too? I think there are some to be deleted that complains on mysql_... [06:42:04] jynus: true, that needs to be changed indeed [06:43:40] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4103233 (10Marostegui) >>! In T54921#4103215, @jcrespo wrote: > Hey, @Marostegui, when deleting tables, can you check views on labs, too? I think there are... [07:20:05] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 2 others: Investigate optimzing wb_terms - https://phabricator.wikimedia.org/T188279#4103329 (10Marostegui) So I have created the following user: `test_user` that has all privileges from terbium. I have already left the p... [07:31:14] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4103375 (10Marostegui) [07:32:26] do I disable then read_only on db2038? [07:32:48] oh, I will do it! [07:32:48] Sorry [07:32:56] done [07:32:59] haha [07:33:04] thanks :) [07:33:52] but it is pooled right now [07:34:01] the whole point was to depool it? [07:34:05] it is depooled [07:34:29] https://noc.wikimedia.org/conf/highlight.php?file=db-codfw.php&5 I don't see that [07:34:34] I know why [07:34:37] You are right, it is not [07:34:41] I deployed db-eqiad [07:34:59] I am too used to deploy db-eqiad instead of db-codfw [07:35:01] Doing it now [07:36:08] done [07:38:16] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4103385 (10Marostegui) For s2, db2041 is a good candidate: same HW and different row. [07:59:25] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4103426 (10Marostegui) [07:59:55] 10DBA: Drop flaggedrevs tables from mediawikiwiki - https://phabricator.wikimedia.org/T186865#4103429 (10Marostegui) 05Open>03Resolved a:03Marostegui This is all done [07:59:58] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4103432 (10Marostegui) [08:12:26] https://mysqlrelease.com/2018/03/mysql-8-0-it-goes-to-11/ [08:14:06] Yeah, I read that a few days ago… [08:14:16] just saying, the logging table on mediawiki is 15M rows, going to delete around 11M-12M of it now [08:14:22] mediawikiwiki [08:14:31] is it fine, won't take much [08:14:57] please log maintenance work in case of issues [08:15:07] that's for sure [08:23:41] This wiki is on s3, lag and stuff is fine fore now [08:24:13] it's weird that didn't even shows much difference in total number of read and writes, is s3 that big? [08:24:54] mediawikiwiki compared to s3 is tiny [08:27:01] around the 3.5% of the total content [08:30:21] https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1075&var-port=9104&from=now-24h&to=now [08:30:27] uptime: 1.97 years [08:30:28] wow [08:32:09] you should have seen the recently upgraded 4y mysql uptime one [08:32:32] Amir1: https://phabricator.wikimedia.org/P6854 [08:33:35] o.O [08:55:33] backups almost finished, they worked nicely [08:55:38] nice!!! [08:55:44] great job! [08:56:16] I am going to do another test restore [09:06:51] tar: Unexpected EOF in archive [09:11:32] oh :( [09:11:37] oh, no it was just me [09:12:46] * jynus breathes with relief [09:13:05] haha [09:13:06] I tried to do "tar -zxf", which strangely [09:13:12] worked partially [09:13:27] instead of tar -xf, as it is a .gz.tar, not a tar.gz [09:13:52] I should just use -xf all the time, as gnu tar autodetects the compression when needed [09:14:30] or, actually [09:14:38] yeah, for me it is almost like autopilot to use zxvf [09:14:39] the recovery hadn't finished yet [09:14:50] more probably [09:19:55] the last backup, m2, of course takes forever [09:23:17] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4103554 (10Marostegui) [09:26:11] oldest backup we have with the new format is from 2018-03-07 [09:27:50] the dates a a bit irregular [09:30:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#4103559 (10Marostegui) [09:31:22] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#4103560 (10Marostegui) [09:37:03] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4103577 (10Marostegui) [09:39:34] As per metada for s1 dump, it is finished, can I run a schema change on codfw with replication enabled or are you planning to do some work that might be affected? [09:40:23] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4103598 (10Ladsgroup) [09:41:22] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4103616 (10Ladsgroup) [09:44:05] sure [09:44:19] thanks :) [09:44:51] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4103634 (10Ladsgroup) [09:44:55] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review, and 2 others: Investigate optimzing wb_terms - https://phabricator.wikimedia.org/T188279#4103632 (10Ladsgroup) 05Open>03Resolved I'd say let's call this done. [10:10:43] 10DBA, 10Schema-change: Schema changes to site_stats - https://phabricator.wikimedia.org/T190780#4103783 (10EddieGP) I can now verify it was applied to beta. Creating pages and editing them still increases the counters on Special:Statistics like it should. There's not much else I can think of that could be tes... [10:20:35] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1020 - https://phabricator.wikimedia.org/T189773#4103875 (10Marostegui) a:05RobH>03Cmjohnson Assigning to Chris to reflect the latest work that was done for this host [10:34:21] 10Blocked-on-schema-change, 10DBA: Schema changes to site_stats - https://phabricator.wikimedia.org/T190780#4103910 (10jcrespo) p:05Triage>03Low > "it does not block any code or further development" It is ok, we can still say it is blocked and give it a lower priority, this is just a list of pending stuff... [10:48:46] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4103945 (10Marostegui) For s3: db2057 is the best candidate. Same HW as the current master and different row. [10:54:34] 10DBA, 10Operations, 10hardware-requests, 10Goal: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#4103958 (10jcrespo) @Cmjohnson and @robh Thanks for all the hard work on eqiad!- once all decommission steps happen (we can and should wait for it to finish, that is mor... [10:56:53] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4002933 (10jcrespo) @Anomie let me refresh my memory- you asked to have testwiki deployed with this change ASAP, is th... [11:47:05] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4104060 (10Marostegui) [12:30:05] backups finished properly, it took 5 hours to backup otrs [13:09:41] 10DBA, 10Operations, 10Patch-For-Review: Firewall configurations for database hosts - https://phabricator.wikimedia.org/T104699#4104380 (10jcrespo) @MoritzMuehlenhoff We should have a full coverage of form on all db and proxy hosts, with the exception of dbproxy1010 and dbproxy1011 that it is managed with th... [13:16:04] jynus: fantastic! I'll doublecheck, but will take a while until I'll get to it [13:17:10] it has been open for 2 years? [13:17:22] I think it can wait... [13:18:35] yeah :-) [13:49:45] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#4104557 (10Marostegui) [13:49:47] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#4104558 (10Marostegui) [13:50:06] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4104559 (10Marostegui) [13:55:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#4104573 (10Marostegui) s1 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1002 [] db1080 []... [13:55:08] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#4104574 (10Marostegui) s1 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db... [13:55:14] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4104575 (10Marostegui) s1 eqiad progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1... [13:55:51] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#4104578 (10Marostegui) [13:55:54] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#4104580 (10Marostegui) [13:56:24] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4104583 (10Marostegui) [14:25:13] marostegui: do you mind leaving db1099:3311 depooled after the alter? I want to do some tests for T186266 [14:25:33] sure! [14:25:48] it will probably be finished by tomorrow or late in the evening [14:26:08] oh [14:26:10] ok [14:26:19] I mean the alter [14:26:27] If you can do your test while the alter runs, that is also fine by me [14:26:39] I actually think I cood [14:26:48] just sync with mee before repooling [14:26:52] will do! [14:28:55] actually, I am finished, the issue doesn't happen on those hosts [14:29:12] so I can repool whenever I am done? [14:32:19] yes [14:32:28] I thought it was going to take more! [14:32:41] sorry [14:32:57] no worries! do you want me to ping you whenever I depool a non rc host from s1? [14:33:01] in case you need to run more tests? [14:33:11] I am going and try to use db2055 [14:33:20] which I think it also has alters ongoing [14:33:29] yeah [14:33:41] all s1 on codfw have alters coming thru replication thread [14:33:49] sorry! :) [14:34:04] (so only finished on enwiki codfw master?) [14:34:10] yep! [14:51:44] 10DBA: Rebuild user_newtalk on db1052 - https://phabricator.wikimedia.org/T186503#3945590 (10jcrespo) Is this really a next? I would downgrade it to low and backlog, with more important stuff happening in the next month. [14:53:12] 10DBA: Rebuild user_newtalk on db1052 - https://phabricator.wikimedia.org/T186503#4104860 (10Marostegui) It is blocked on the dc failover, so I am fine if we move it to backlog [15:10:51] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4104965 (10jcrespo) This is blocked on @Cmjohnson to have a gap for firmware and BIOS upgrade + RAID rebuild as asked here T18659... [15:15:42] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T191275#4104995 (10Marostegui) For s4 I suggest: db2058 Same hardware as the master and it is in a different row. [15:19:13] I am going to start a backup of es2015 into es2002, emphasis on "start" [15:21:42] 10DBA, 10Operations: Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave - https://phabricator.wikimedia.org/T153440#4105044 (10jcrespo) a:03jcrespo I am going to start doing some tests onto es2002. [15:28:12] it creates lag, so I will do a depool [15:51:10] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4105185 (10Anomie) We did that for the comment table change, but then it turned out that CentralAuth broke if we turne... [16:01:26] 10DBA, 10Operations, 10Patch-For-Review: Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave - https://phabricator.wikimedia.org/T153440#4105257 (10jcrespo) [16:10:10] 10DBA, 10MediaWiki-Platform-Team, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4105300 (10Anomie) This won't be ready to go until (probably) the end week, it's blocked on T191307 which I plan to finish on Friday. I left off #blocked-on-schema-... [16:15:03] 10DBA, 10MediaWiki-Platform-Team, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4105309 (10Marostegui) Ah cool! Thanks for the clarification! I'm hoping to finish with T185128 later next week or so. So I will start working on this schema change... [16:16:40] 10DBA, 10MediaWiki-Database, 10MediaWiki-Special-pages, 10Security, 10Wikimedia-log-errors: Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way. - https://phabricator.wikimedia.org/T191116#4105327 (10Anomie) Looks like Aaron already fixed that one too: {2627206}. Let me know... [16:18:11] 10DBA, 10MediaWiki-Database, 10MediaWiki-Special-pages, 10Security, 10Wikimedia-log-errors: Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way. - https://phabricator.wikimedia.org/T191116#4105330 (10Marostegui) We can wait for it to be gone tomorrow :) Thanks! [16:25:49] 10DBA, 10Operations, 10Patch-For-Review: Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave - https://phabricator.wikimedia.org/T153440#4105354 (10jcrespo) @ayounsi We are performing this 5.4TB backup by saturating the es2015 and es2002 host lin... [16:42:21] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4105469 (10RobH) I started the idrac firmware update, it is taking 5+ minutes to update. when done, it should show version 2.52 f... [17:22:56] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4105730 (10RobH) I neglected to note the old bios and drac versions, but they are now latest versions each and done. h710 mini co... [17:24:04] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4105743 (10RobH) @jcrespo: Did you want to handle the raid rebuild? I'm not exactly sure what you want? (Just to wipe it all out... [17:32:22] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-notice, and 2 others: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4105808 (10mmodell) [18:51:04] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4106022 (10CCicalese_WMF) [18:55:52] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4106040 (10CCicalese_WMF) [18:55:55] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#4106041 (10CCicalese_WMF) [19:05:05] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1020 - https://phabricator.wikimedia.org/T189773#4106160 (10Cmjohnson) [19:05:16] 10DBA, 10Operations, 10hardware-requests, 10Goal: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#4106165 (10Cmjohnson) [19:05:19] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4106164 (10Cmjohnson) [19:05:21] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1020 - https://phabricator.wikimedia.org/T189773#4052908 (10Cmjohnson) 05Open>03Resolved [19:11:16] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4106213 (10CCicalese_WMF) [21:08:34] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-notice, and 2 others: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4106799 (10aaron)...