[09:32:51] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) The comparison finished, and the data is OK. [10:40:41] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:58:34] happy new year all :) [10:59:28] happy new year indeed [11:00:58] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10jcrespo) I would like to insist on this issue now that the holiday is over- while the service (parsercache) is not at the time affected, we are in a no-hw redundancy mode on eqiad, and after all... [15:07:17] I did the sanitization for testcommonswiki, now I am re-running the check_private_data.py script to see if it was successful. [15:20:19] great [15:29:26] I am a bit behind because I am still checking the context of that wiki creation [15:30:39] https://phabricator.wikimedia.org/T197616#4832209 [15:30:50] Apparently it should be going away later this year [15:46:14] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10jcrespo) I agree with Manuel T197616, I would have preferred creating it on s3 for isolation reasons- enwiki, commons and w... [15:49:15] I commented there before I saw the comment [15:49:20] ŷour [15:49:21] here [16:13:40] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T212966 (10Papaul) a:05Papaul→03Marostegui Disk replacement complete [16:18:25] jynus: Check in with JamesF, but there was lots of assumptions in the (mw?) config about commons == s4 which would've required a non trivial amount of work for this test wiki to exist too [16:18:36] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Jdforrester-WMF) >>! In T197616#4859391, @jcrespo wrote: > Based on comments that removing the wiki is going to happen, I w... [16:19:00] Or... So was said on IRC [16:19:13] I would have a lot to say about that, but not the time :-D [16:19:50] Reedy: actually, Jf just said the opposite on phab :-P [16:19:57] Yeah... I'm a little confused [16:20:01] he he [16:20:10] But just puttig it in s3 on the mw-config patch broke stuff [16:20:27] And resulted in ariel saying something like "nope, I'm not getting nerd sniped by this. I'm going to bed" [16:20:34] as I said, is it has an expery date, I prefer to not touch it [16:20:40] *as [16:22:30] I prefer to actually make sure it is deleted at a later time, rather than make you work more on that [16:24:14] Please feel free to hold us to the task of deleting and cleaning up after it later :) [16:24:40] I WILL :-P [16:25:00] * Reedy prepares the "not yet" comments for when things are inevitably delayed and need to be pushed back [16:25:26] it is ok, I know delays will happen [16:25:30] I am ok with that [16:25:54] I just want to guilt people into making sure it happens eventually 0:-) [16:26:22] I am more worried about "s4 is commons" (or the other way) [16:26:36] because that is a mistake ready to happen [16:26:54] we can help, we did the s5 migration quite panlesly [16:27:55] James just said he'll file a task about the s4 == commons issues [16:28:08] and please not that I khow I am complaining to the person that actually did something. sorry about taht [16:28:37] I created the wiki [16:28:40] So I did do something :P [16:28:56] *know [16:29:02] note* [16:29:13] I am trying to say thank you :-) [16:29:21] but also attending an outage at the same time [16:29:33] yeah, deal with that :D [16:53:34] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Marostegui) I wonder why was this created in s4 if we were asked if we preferred s3 and I replied a day after that question... [16:56:25] 10DBA, 10Analytics, 10Analytics-Cluster, 10Operations: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Milimetric) @Dzahn wikimetrics is going to be sunset this quarter, so you won't have to worry about that any more. [16:56:48] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10jcrespo) I've been told there was some breakage based on assuming s4 ==> commons, or commons ==> s4. I am not too worried,... [16:57:18] 10DBA, 10Analytics, 10Analytics-Cluster, 10Operations: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Ottomata) OK FINE I'LL DO IT [16:57:29] 10DBA, 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Milimetric) a:03Ottomata [16:57:37] 10DBA, 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Milimetric) p:05Normal→03High [16:58:33] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Jdforrester-WMF) >>! In T197616#4859717, @Marostegui wrote: > I wonder why was this created in s4 if we were asked if we pr... [16:59:05] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Reedy) >>! In T197616#4859717, @Marostegui wrote: > I wonder why was this created in s4 if we were asked if we preferred s3... [17:09:16] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Marostegui) Waiting for one hour only is not realistic, specially if asked at 19:00 UTC. The 4th I was on holidays (I am st... [17:13:44] https://phabricator.wikimedia.org/T212909 [17:13:45] https://phabricator.wikimedia.org/T212910 [17:13:48] https://phabricator.wikimedia.org/T212861 [17:15:32] I totally forgot those [17:38:02] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) @jcrespo An email was sent to Dell requesting a new board. I have not received a response [17:54:09] How do we sql onto es boxes these days? [17:54:44] es? [17:54:57] it should be the usal way [17:56:03] check the master and just login there or a particular host [17:57:11] previously you could do sql dbname -h ipaddress [17:57:23] but that looks to have been broken at some point [17:58:01] Or maybe just a change of syntax [17:58:01] sql testcommonswiki -- -h 10.64.32.184 [17:58:02] and I heard it got fixed? [17:58:17] but not sure if with the same syntax [17:58:23] check the history [17:58:35] you probably won't get a lot of documentation :-) [17:58:49] do you need anything in particular? [17:59:10] I'm trying to work out why MW is bitching about the ES tables not existing for testcommonswiki [17:59:18] Error: 1146 Table 'testcommonswiki.blobs_cluster24' doesn't exist (10.64.32.184) [17:59:43] that is an sql error, so probably right [17:59:52] let me check [18:00:22] `sql testcommonswiki -- -h 10.64.32.184` gives a timeout [18:01:52] the db is there but not the table [18:02:10] same on es3 [18:02:25] I would expect a cluster25 there [18:02:29] indeed [18:02:45] so the table creation failed or was skipped [18:02:58] We did get https://phabricator.wikimedia.org/T212881 [18:02:59] it is not a permission or something [18:03:11] I don't know if it's due to some recent changes to mw/addWiki.php by Aaron [18:03:21] Or related to dbstore1002 being crappy [18:03:23] dbstore1002 is not related [18:03:33] it is not mediawiki and to be decommed [18:03:39] not part of production [18:04:02] and mediawiki doesn't know about it [18:04:20] are you getting that right now? [18:04:36] Nope [18:04:43] then retry? [18:04:54] I'm not re-running addWiki because it's not very... "safe" [18:05:00] oh [18:05:08] I thought there was a dedicated script [18:05:13] what failed [18:05:17] so addwiki failed [18:05:28] maybe because some unrelated issue [18:05:38] Like I say, it might be due to recent MW refactorings [18:05:39] I mean, I can create those manually [18:05:48] That, and every time we run addWiki.php it's broken [18:05:49] if that helps, but will fail later [18:05:54] Yeah, if you wouldn't mind, that unblocks us for now [18:06:03] When we next run addWiki we can see if it breaks again [18:06:04] ok, doing, will log on operations [18:06:15] Thanks [18:06:25] This commit seems slightly suspicious [18:06:25] https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/commit/c0569fb89a449d86c5ad5aad5a178ac109de96df [18:06:37] so, just to be clear, dbstore1002 issues are unrelated [18:06:47] That's fine [18:06:58] Just after the script failing, dbstore1002 started giving replication errors [18:07:05] well, replag, not replication errors [18:07:13] It just seemed a bit suspicious at the time :) [18:07:14] yes, but it would have happened too if it hadn't failed [18:07:28] it is because how multisource works + tokudb [18:07:48] we are removing that and setting up new host like the ones on production that will fix that [18:08:16] and actually stop using toku db and multisorce everywhere except on labs [18:12:23] Reedy: I think it is done now [18:12:45] Thanks! [18:12:50] Yeah, editing isn't completely broken on it now [18:12:56] completely? [18:13:17] honestly, probably the first page that is automatically created was lost [18:13:21] (the homepage) [18:13:47] I don't know if any more content is created automatically [18:13:59] Nope, only that one by default [18:14:05] That being lost isn't the end of the world [18:29:46] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [18:36:10] 10DBA, 10Data-Services, 10Operations: db1082 power loss resulted on mysql crash - https://phabricator.wikimedia.org/T213108 (10jcrespo) p:05Triage→03High [18:37:06] 10DBA, 10Data-Services, 10Operations: db1082 power loss resulted on mysql crash - https://phabricator.wikimedia.org/T213108 (10jcrespo) a:05Cmjohnson→03jcrespo I plan to take care of this tomorrow morning. [20:16:10] 10DBA, 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, and 2 others: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Ottomata) @Dzahn https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/482693/ [21:03:50] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Neil_P._Quinn_WMF) >>! In T212487#4839932, @elukey wrote: > `datasets` seems indeed not us... [22:54:31] 10DBA, 10Data-Services, 10Operations: db1082 power loss resulted on mysql crash - https://phabricator.wikimedia.org/T213108 (10Marostegui) Maybe it is worth to start replication on db1082 (not on sanitarium), let it catch up, once it is synced compare.py it against the host you will reimage it from to make s...