[05:32:29] 10DBA, 10cloud-services-team: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3488463 (10Marostegui) [06:32:53] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3488497 (10Marostegui) [08:25:00] 10DBA, 10MediaWiki-Watchlist, 10Wikimedia-General-or-Unknown, 10Wikimedia-log-errors: User with 40000 entries in their Watchlist cannot access it on Commons anymore: Database error - https://phabricator.wikimedia.org/T171898#3488601 (10Jwh) >>! In T171898#3481750, @Aklapper wrote: > Workaround: You could a... [08:37:50] 10DBA: db2019 has performance issues, replace disk or switchover s4 master elsewhere - https://phabricator.wikimedia.org/T170351#3488615 (10Marostegui) As part of: T170662 we will probably switch over s4 master on codfw. So this will get resolved. I was thinking about either db2051 or db2065 to replace the curre... [08:47:10] 10DBA: db2019 has performance issues, replace disk or switchover s4 master elsewhere - https://phabricator.wikimedia.org/T170351#3488638 (10jcrespo) db2051 is probably the right one, but we need to spread that hw batch, as you suggested, first. [08:50:32] 10DBA, 10Operations, 10ops-codfw: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3488647 (10Marostegui) Hi @Papaul We are planning to switchover db2019, and we will promote db2051 to master which is in C6, but we'd like to move it to B8 for instance? Could you let us know which... [09:11:56] 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 4 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3488734 (10Addshore) [09:24:10] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10Wikimedia-maintenance-script-run: Unbreak/finish global merge of Yuvipanda into YuviPanda - https://phabricator.wikimedia.org/T104686#3488739 (10MarcoAurelio) p:05High>03Unbreak! a:05Legoktm>03None It is blocking editting from this editor and is run... [09:54:26] 10DBA, 10MediaWiki-History-or-Diffs, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) - https://phabricator.wikimedia.org/T163681#3488759 (10Dereckson) When we profile a request, we see a request as... [10:26:59] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3488793 (10thiemowmde) [12:14:09] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Community-Tech, 10User-Elukey: Drop CookieBlock* tables from EventLogging DB - https://phabricator.wikimedia.org/T171883#3488967 (10elukey) Action executed on db1046 (m4-master): ``` MariaDB [(none)]> use log; Reading table information for completio... [12:51:20] 10DBA, 10cloud-services-team: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3489024 (10Marostegui) [14:38:16] 10DBA, 10Operations, 10ops-codfw: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3489286 (10Papaul) Hi @Marostegui We can do this tomorrow with no problem. [14:38:40] 10DBA, 10Operations, 10ops-codfw: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3489287 (10Marostegui) >>! In T169501#3489286, @Papaul wrote: > Hi @Marostegui > > We can do this tomorrow with no problem. Sounds good, any specific time? [15:05:45] 10DBA, 10Operations, 10ops-codfw: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3489466 (10Papaul) 10:00 am CDT [15:07:45] 10DBA, 10Operations, 10ops-codfw: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3489472 (10Marostegui) Excellent, I will get the server ready by then, just to change the IP on it and power it off :) Thanks a lot! [15:08:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3489474 (10Marostegui) db1055 is done: ``` root@neodymium:/home/marostegui# for i in `cat s1_tables`; do echo $i; mysql --skip-ssl -hdb1055 enwik... [15:09:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3489480 (10Marostegui) [15:21:21] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10Wikimedia-maintenance-script-run: Unbreak/finish global merge of Yuvipanda into YuviPanda - https://phabricator.wikimedia.org/T104686#3489517 (10Jdforrester-WMF) p:05Unbreak!>03High This affects one account, and is not a risk to life or limb, nor are th... [15:58:56] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3489609 (10Nuria) @Neil_P._Quinn_WMF The labs snapshot (with newer wikis but not all) is about to start, all things going well it will take 3/4 days for it... [16:10:31] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3489668 (10Milimetric) Ok, I just deployed the list for all but the 12 wikis that are still imported. That means the next reconstructio... [18:00:07] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490055 (10Reedy) [18:06:44] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490072 (10jcrespo) From what I can see it is not a mistake on ops side (which wouldn't be that rare)- it is not on the code either (if I am not wrong): ht... [18:08:06] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490076 (10Reedy) I'm guessing there's not going to be many merges in the future, but I may be wrong. And of course, it's only an issue if one of the wiki... [18:09:56] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490082 (10jcrespo) > I'm guessing there's not going to be many merges in the future, but I may be wrong. Yes, I am not indicating the priority of this, j... [18:11:25] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490037 (10Marostegui) >>! In T172207#3490076, @Reedy wrote: > > > And of course, it's only an issue if one of the wikis that a merge needs to happen has... [18:12:17] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490106 (10Reedy) List of wikis running flagged revs? We have a dblist :) https://noc.wikimedia.org/conf/highlight.php?file=flaggedrevs.dblist [18:13:33] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490110 (10Marostegui) >>! In T172207#3490106, @Reedy wrote: > List of wikis running flagged revs? We have a dblist :) > > https://noc.wikimedia.org/conf/... [18:29:30] 10DBA, 10MediaWiki-Watchlist, 10Wikimedia-General-or-Unknown, 10Wikimedia-log-errors: User with 40000 entries in their Watchlist cannot access it on Commons anymore: Database error - https://phabricator.wikimedia.org/T171898#3479604 (10jcrespo) There is not much we can do here on persistenace layer- Mediaw... [18:55:02] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490037 (10MarcoAurelio) Dupe {T105398}? [18:57:15] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490314 (10Reedy) Looks like it, but we've come across it again. Adding it to the extension is trivial in MW land, getting it deployed is harder. Do we j... [19:01:54] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490361 (10MarcoAurelio) I don't know how trivial or hard would be. I know no coding. However if it causes issues we should get this eventually done. Would... [19:07:30] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490425 (10MarcoAurelio) [19:09:52] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490451 (10Reedy) Nope, nothing like it. Though, I just noticed it does at least have a primary key, so the DBA's may have an easier jo... [19:17:46] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490514 (10MarcoAurelio) Yep. For reference "only" [[ https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/dblists/f... [19:25:27] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490561 (10Reedy) >>! In T172207#3490514, @MarcoAurelio wrote: > Yep. For reference "only" [[ https://phabricator... [19:39:47] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490602 (10MarcoAurelio) > Nothing needs populating Ah, okay. I though we need populating. Thanks :) [19:47:41] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490625 (10Marostegui) >>! In T172207#3490514, @MarcoAurelio wrote: > Yep. For reference "only" [[ https://phabri... [19:55:07] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490659 (10Reedy) Most wikis won't have the tables. There's probably a few that had flagged revs installed, but it... [20:04:18] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490037 (10demon) More like just shy of 6% ``` chad@notsexy /a/ops/mediawiki-config/dblists (master)$ cat flaggedr... [22:17:53] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3491226 (10Ladsgroup) [22:17:58] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3491228 (10Ladsgroup) [23:47:57] 10DBA, 10MediaWiki-extensions-Babel, 10Wikimedia-General-or-Unknown: Create and populate babel database table on Wikimedia wikis - https://phabricator.wikimedia.org/T145366#3491572 (10Reedy) [23:48:03] 10DBA, 10MediaWiki-extensions-Babel, 10Wikimedia-General-or-Unknown: Create and populate babel database table on Wikimedia wikis - https://phabricator.wikimedia.org/T145366#2627711 (10Reedy) [23:48:10] 10DBA, 10MediaWiki-extensions-Babel, 10Wikimedia-General-or-Unknown: Create and populate babel database table on Wikimedia wikis - https://phabricator.wikimedia.org/T145366#2627711 (10Reedy) 05Open>03Resolved a:03Reedy