[00:05:02] 10Blocked-on-schema-change, 10Community-Tech, 10MediaWiki-Database, 10Hindi-Sites, and 3 others: Allow comments longer than 255 bytes - https://phabricator.wikimedia.org/T6715#3491636 (10tstarling) [08:19:18] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117#1685086 (10Multichill) Ran into this one. Take for example https://commons.wikimedia.org/wiki/File:Armor_of_Emperor_Ferdinand_I_(1503%E2%80%931564)_MET_DP-1288... [08:42:16] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3492207 (10Marostegui) `click_tracking` table has been dropped. [08:42:25] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3492208 (10Marostegui) [09:40:41] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3492349 (10Marostegui) [09:40:45] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3492346 (10Marostegui) 05Open>03Resolved a:03Marostegui `click_tracking_user_pro... [09:40:49] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3492350 (10Marostegui) [10:20:18] 10DBA, 10Reading-Web-Backlog (Tracking): Drop WikiGrok tables from production - https://phabricator.wikimedia.org/T172020#3492429 (10Marostegui) a:03Marostegui This tables exists on: s1: enwiki s3: test2wiki, testwiki, testwikidatawiki (mostly empty tables) s5: wikidatawiki I have backup'ed them on: ``` ro... [10:27:29] marostegui: do you have some time to have a look at https://gerrit.wikimedia.org/r/#/c/369397/ [10:27:51] Yep! I will in a bit [10:58:11] 10DBA, 10Operations, 10ops-codfw: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3492537 (10Marostegui) [11:04:43] 10DBA, 10Operations, 10ops-codfw: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3492564 (10Marostegui) For the record, after manually forcing the re-learn we got it back to healthy - let's see how long until it fails again: ``` ˜/icinga-wm 13:03> RECOVERY - MegaRAID on es2013 is OK: OK: optim... [11:06:18] https://gerrit.wikimedia.org/r/#/c/369629/ [11:06:33] +1'ed [11:09:29] there is a very amusing untracked file on mediawiki-staging [11:10:12] yeah [11:10:17] I pinged Reedy in the morning [11:10:21] When I saw it :) [11:10:37] oh, not so amusing, I thought it was a typo [11:10:46] it is actually a thing [11:10:50] it is a log [11:10:52] or looks like a log [11:11:52] yes, it is the scap log [11:12:13] yep: ˜/marostegui 8:51> Reedy: There is: tin:/srv/mediawiki-staging/wikinieuws owned by you, which looks like a log from a deployment of InitialiseSettings.php (not a big deal, but just letting you know) [11:12:17] XDDD [11:12:25] He might be in another TZ :) [11:26:52] https://phabricator.wikimedia.org/P5843 [11:31:04] Oh, so 3 of them…maybe those were involved in HW replacements or stuff? [11:34:08] should I do the same on db[1096-1106], db[2071-2092] ? [11:34:27] I would say so if you have time [11:34:32] just to be on the safe side [11:34:42] it might be enabled on all the new hosts by default [11:34:46] I am asking because I am only 99% percet this is the right thing to do [11:34:50] At least on the analytics new hosts it was enabled [11:35:04] So it is likely it will be there too [11:35:08] because later we forget about running learning cycles and the bbu goes bad [11:35:12] yeah [11:35:17] which is equaly bad or worse [11:37:38] cumin 'db[2071-2092]' 'echo "autoLearnMode=1" > /home/jynus/disableAutoLearn.txt && megacli -AdpBbuCmd -SetBbuProperties -f /home/jynus/disableAutoLearn.txt -aALL' [11:37:43] ^please review [11:37:54] checking [11:37:55] will run it on codfw first, then on eqiad [11:38:07] yeah, that looks right [11:38:24] I am afrad of doing an rm -f, sorry [11:38:30] :-) [11:38:35] haha [11:39:30] I will add disabling auto-learn to the raid setup instructions [11:39:41] at least for documentation [11:39:54] good idea yeah [11:43:29] fun fact, db1047 has two raid devices- and autolearn is enabled in one and disabled on the other [11:45:40] I think we had issues with one of them, and maybe disabled only one [11:45:46] I always forget it has two :) [11:57:22] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3492726 (10Marostegui) [12:05:27] just FYI cumin accepts multiple commands and they are executed only if the first succeed, based on the mode chosen, the same would be: [12:05:44] cumin -m async 'hosts...' 'command1' 'command2' ... [12:05:47] * marostegui confirms that volans has a "cumin" highlight [12:06:01] see https://wikitech.wikimedia.org/wiki/Cumin#Command_execution for a detailed description of the 2 available modes ;0 [12:06:04] ;) [12:06:28] marostegui: as always, I cannot confirm nor deny, I keep an eye on this channel, you know :-P [12:06:43] hahaha [12:08:12] marostegui: Lol. Yeah, that wants deleting [12:08:32] gone [12:08:41] \o/ [12:08:46] thank you! [12:59:51] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490037 (10Luke081515) btw, what does flaggedrevs_labswikimedia do? I don't think we have flaggedrevs at any labsw... [13:02:22] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493109 (10Marostegui) >>! In T172207#3493104, @Luke081515 wrote: > btw, what does flaggedrevs_labswikimedia do? I... [13:05:07] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493117 (10Reedy) `flaggedrevs_labswikimedia` is a deleted wiki. It predates beta et al, and presumably was a test... [13:10:45] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493159 (10jcrespo) I would spent time dropping the table on those places where the extension is disabled instead... [13:26:42] the daily terbium purging is probably taking close to 24 hours [13:26:50] (of pc* hosts) [13:29:04] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3493190 (10jcrespo) This is the one-liner I am thinking of cronify on parsercaches so we can forget about this task, even if there is in the future the... [13:29:29] although my purging may make things slower [13:29:34] *defragmenting [13:33:43] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493218 (10Reedy) A quick look at https://github.com/wikimedia/operations-mediawiki-config/commits/master/dblists/... [13:35:54] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493241 (10jcrespo) :-) [14:00:07] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3493377 (10Marostegui) >>! In T167784#3493190, @jcrespo wrote: > This is the one-liner I am thinking of cronify on parsercaches so we can forget about... [14:00:39] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493379 (10Marostegui) >>! In T172207#3493218, @Reedy wrote: > A quick look at https://github.com/wikimedia/operat... [14:05:21] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10Patch-For-Review, 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3493399 (10Reedy) I guess if we compare that dblist and the list you added to the description of what files on dis... [14:25:43] 10DBA, 10Data-Services: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493503 (10zhuyifei1999) [14:27:03] 10DBA, 10Data-Services: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#1455384 (10Marostegui) @marcmiquel is this still an issue? [14:43:20] 10DBA, 10Data-Services: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493551 (10marcmiquel) Nope, closed. Thank you În Mie, 2 aug. 2017, 16:27 Marostegui, a... [14:45:47] 10DBA, 10Data-Services: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493560 (10zhuyifei1999) 05Open>03Resolved [15:13:11] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Move some masters away from B6 - https://phabricator.wikimedia.org/T169501#3493629 (10Papaul) New switch port configuration for db2051 asw-b8-codfw ge-8/0/17 [16:12:00] 10DBA, 10Patch-For-Review: db2019 has performance issues, replace disk or switchover s4 master elsewhere - https://phabricator.wikimedia.org/T170351#3493783 (10Marostegui) db2051 has been moved and it is now replicating again from its new location Thanks @papaul! [18:53:22] 10DBA, 10ProofreadPage, 10MW-1.30-release-notes (WMF-deploy-2017-05-23_(1.30.0-wmf.2)), 10Patch-For-Review: Implements ContentHandler abstraction for ProofreadPage Index: pages - https://phabricator.wikimedia.org/T161524#3494559 (10Tpt) 05Open>03Resolved a:03Tpt Done [18:56:41] 10DBA, 10Data-Services, 10Security-Team, 10WMF-Legal, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3494567 (10Bawolff) I approve on behalf of #security-team [19:59:37] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3494814 (10Krinkle) [19:59:48] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3224448 (10Krinkle) [23:00:28] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3495451 (10Ladsgroup) [23:11:12] 10DBA, 10Data-Services, 10Security-Team, 10WMF-Legal, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3495469 (10bd808) a:03madhuvishy @madhuvishy the patch at https://gerrit.wikimedia.org/r/#/c/365969/3 is ready for your review and merge f...