[01:19:49] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10Serbian-Sites, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Liuxinyu970226) [06:25:05] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) s5 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [x] dbstore1002 [] db1124 [x] db1113 [x] db1110 [x] db1102 [x] db1100 [x] db1097 [x] db109... [06:31:49] 10DBA, 10MediaWiki-API, 10MediaWiki-Database: prop=revisions API timing out for a specific user and pages they edited - https://phabricator.wikimedia.org/T197486 (10Marostegui) 05Open→03Resolved a:03Marostegui All the core replicas that receive this query are now running > 10.1.36 which doesn't have th... [07:35:24] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342 (10Marostegui) [07:43:32] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Daimona) @Marostegui Thanks! I tried several different queries on dewiki keeping an eye on Tendril, and nothing suspicious showed up, so I think you can safely go on. I'll contin... [08:00:48] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) Thanks @Daimona! Is this an example of a slow query? ` SELECT /* IndexPager::buildQueryInfo (AbuseLogPager) xxx */ * FROM `abuse_filter_log` LEFT JOIN `abuse_filter`... [08:05:37] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) [08:06:10] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342 (10Marostegui) s6 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [x] dbstore1002 [x] dbstore1001 [] db1125 [] db1113 [] db1098 [] db10... [08:09:59] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) [08:47:54] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) s6 eqiad progress [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1005 [x] dbstore1002 [x] dbstore1001 [x] db1125 [x] db1113 [x] db1098 [x] db1096 [x] db1093... [08:48:11] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) [08:57:55] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) [09:16:03] 10DBA, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Increase parsercache keys TTL from 22 days back to 30 days - https://phabricator.wikimedia.org/T210992 (10Marostegui) I have increased the TTL back to 30 days. Going to monitor the graphs for a few days before closing this. [09:16:41] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Daimona) >>! In T187295#4979679, @Marostegui wrote: > Thanks @Daimona! > Is this an example of a slow query? > ` > SELECT /* IndexPager::buildQueryInfo (AbuseLogPager) xxx */ * F... [09:17:32] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) >>! In T187295#4979911, @Daimona wrote: >>>! In T187295#4979679, @Marostegui wrote: >> Thanks @Daimona! >> Is this an example of a slow query? >> ` >> SELECT /* Index... [10:12:16] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) s2 eqiad progress [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1004 [x] dbstore1002 [x] db1125 [x] db1122 [x] db1105 [x] db1103 [x] db1095 [x] db1090 [x] d... [10:12:46] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) [10:14:44] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) @Daimona the following wikis are now fully altered, I am going to give it some hours before continuing and will monitor tendril: s2: bgwiki bgwiktionary cswiki enwiki... [10:37:09] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Daimona) >>! In T187295#4979912, @Marostegui wrote: >>>! In T187295#4979911, @Daimona wrote: >>>>! In T187295#4979679, @Marostegui wrote: >>> Thanks @Daimona! >>> Is this an exam... [11:16:04] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10jcrespo) Not closed because needs documentation (but it is done). [11:17:10] 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo) [11:17:24] 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo) [11:18:11] 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo) [16:01:10] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Daimona) Nothing weird seems to have happened during the last 6 hours, and this is good. However, I guess that any possible trouble won't be visible before reaching enwiki... [16:03:13] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) >>! In T187295#4981381, @Daimona wrote: > Nothing weird seems to have happened during the last 6 hours, and this is good. However, I guess that any possible trouble w... [16:05:55] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Daimona) Great! Let me know if I can help, e.g. by playing around with AF on enwiki. [16:07:38] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 (10Marostegui) Yeah, I will ask for that once we have more replicas altered, otherwise you might sometimes reach one that is altered and another time one that is not :) Thank you! [16:32:11] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 crashed (HW memory issues) - https://phabricator.wikimedia.org/T214720 (10Cmjohnson) This will most like need a new motherboard. I requested one through Dell You have successfully submitted request SR986942076. [16:32:48] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 crashed (HW memory issues) - https://phabricator.wikimedia.org/T214720 (10Marostegui) Excellent! Thank you Chris! [18:44:00] Remember T136425? That was a few years ago, and now orphaned echo_event rows have built up to about the same level (60M out of 90M on enwiki). I'd like to run the cleanup script again, please let me know when is a good time to do that [18:44:01] T136425: Remove orphaned echo_event rows - https://phabricator.wikimedia.org/T136425 [18:44:38] (This script will delete something like 100M-200M old rows from the Echo databases on x1) [18:46:57] does it have to run soon? I would wait for wednesday just after the backup, just in case [18:47:38] not because I don't trust the script, anything that deletes a lot of stuff is more safely run with fresh backups :-) [18:48:01] No rush at all, the last time we ran it was 2.5 years ago, so 2 more days is nothing [18:48:17] regarding time, any time we are around, at least at the start [18:48:23] how early do you get up? :-) [18:48:48] 15 minutes before my first meeting usually :P [18:48:57] also put it on the Deployments page, there is a "week of" section for that [18:49:28] But on Wednesday my first meeting is at 9am Pacific (18:00 CEST), so I can kick it off around that time [18:49:36] if it frees a lot of space it may be nice to compact the tables to free actual space [18:49:44] *disk [18:49:57] so I would say either at that time [18:50:16] or late in your day so we can monitor it in our morning, as you prefer [18:51:41] 10DBA, 10Growth-Team, 10Notifications, 10WorkType-Maintenance: Clean up orphaned echo_event rows again - https://phabricator.wikimedia.org/T217073 (10Catrope) [18:51:58] 10DBA, 10Notifications, 10Growth-Team (Current Sprint), 10WorkType-Maintenance: Clean up orphaned echo_event rows again - https://phabricator.wikimedia.org/T217073 (10Catrope) a:03Catrope [18:52:50] also fresh backups will allow to compare how much actual data was deleted [18:53:13] "Good news", I also have a meeting 22:00-23:00 my time, which is 07:00-08:00 for you [18:53:26] both work [18:53:44] I tend to be late, manuel tends to be early [18:54:18] so anytime except the 19-6 UTC band [18:54:36] I think I'll go with my morning / your evening, that way I don't hit enter on the script and then immediately go to bed [18:54:50] So 17:00ish UTC [18:54:59] ok [18:56:10] 10DBA, 10Notifications, 10Growth-Team (Current Sprint), 10WorkType-Maintenance: Clean up orphaned echo_event rows again - https://phabricator.wikimedia.org/T217073 (10jcrespo) ^Something to keep an eye on, I asked to be documented on Deployments wiki page and suggested Wednesday, the day after the Monday-t... [18:58:12] 10DBA, 10Notifications, 10Growth-Team (Current Sprint), 10WorkType-Maintenance: Clean up orphaned echo_event rows again - https://phabricator.wikimedia.org/T217073 (10Catrope) I've set a reminder to kick this off at 17:00 UTC on Wednesday February 27. [20:09:56] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601 (10Surlycyborg) [21:54:15] so, I'm trying to rebuild the beta cluster databases and I keep seeing reference to innobackupex + xtrabackup but they don't seem to be installed nor are they available in apt. Are they packaged? [22:00:58] 10DBA, 10Wikimedia-Site-requests, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) @Marostegui @jcrespo - green light to retry the rename aft... [22:16:13] twentyafterfour: percona toolkit? [22:32:31] Reedy: I don't know, /me tries it [22:32:51] it looks to my like deployment-db05 is healthy [22:33:01] I just need to figure out how to make that the db master [22:35:00] hmm percona-toolkit doesn't seem to have it [22:40:06] percona-xtrabackup [22:52:29] I don't actually think I need it (apparently the data is there) [23:31:54] !log root@deployment-db05# mariabackup --innobackupex --apply-log --use-memory=10G /srv/sqldata # T216067 [23:31:54] twentyafterfour: Not expecting to hear !log here [23:31:55] T216067: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067