[01:36:00] 10DBA, 10Expiring-Watchlist-Items, 10Community-Tech (Kanban-Q3-2019-20), 10Core Platform Team Workboards (Clinic Duty Team): Create required table for new Watchlist Expiry feature - https://phabricator.wikimedia.org/T240094 (10Mooeypoo) Thank you for the quick response, @Marostegui. >>! In T240094#5825657... [07:24:06] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Patch-For-Review, and 2 others: Add index to recentchanges.rc_source - https://phabricator.wikimedia.org/T244080 (10Marostegui) >>! In T244080#5843891, @DannyS712 wrote: >>>! In T244080#5843882, @Krinkle wrote: >> @DannyS712 Can you provide an example of... [08:11:57] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) [08:17:16] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) p:05Triage→03Normal [08:50:12] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1139 [] db1134 [] db1124 [] db1119 [] db1118 [] db1107 []... [08:50:39] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) [09:05:26] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) [09:05:34] 10Blocked-on-schema-change, 10DBA, 10User-DannyS712: Apply: Add primary key to pagetriage_page_tags - https://phabricator.wikimedia.org/T243804 (10Marostegui) 05Open→03Resolved All done [09:35:18] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [09:41:52] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) [09:42:06] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) p:05Triage→03Normal [11:06:02] 10DBA, 10Expiring-Watchlist-Items, 10Community-Tech (Kanban-Q3-2019-20), 10Core Platform Team Workboards (Clinic Duty Team): Create required table for new Watchlist Expiry feature - https://phabricator.wikimedia.org/T240094 (10Marostegui) >>! In T240094#5846792, @Mooeypoo wrote: > > We are flexible; we in... [11:21:05] marostegui: ^there was an IRC netsplit, we were almost the only ones isolated on one side, all others on other servers [11:21:17] haha [11:21:21] destiny! [11:21:33] well, at least we could communicate [11:21:41] I don't have join/part messages, so I didn't notice [11:22:29] see how log didn't get our conv: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-databases/20200204.txt [11:22:45] I see [11:22:57] they can be annoying at times, but useful at other times [14:19:33] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Andrew) For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it in my morning (e.g. 15:00 UTC) then I can send... [14:24:44] 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10ayounsi) Anytime works for LibreNMS. [14:24:56] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) >>! In T244209#5848154, @Andrew wrote: > For a few seconds interruption I wouldn't expect this to be very disruptive. If you schedule it i... [14:25:14] 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Marostegui) >>! In T244238#5848193, @ayounsi wrote: > Anytime works for LibreNMS. Thank you! <3 [14:27:18] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Andrew) >>! In T244209#5848195, @Marostegui wrote: > > Thank you! > What about Monday 10th at 15:00 UTC? Works for me! I'll put it on the team calen... [14:28:01] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) >>! In T244209#5848200, @Andrew wrote: >>>! In T244209#5848195, @Marostegui wrote: >> >> Thank you! >> What about Monday 10th at 15:00 UTC... [14:33:53] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Andrew) I'll do it now. [14:39:41] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Marostegui) Thank you! [14:40:51] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Upgrade and restart m5 master (db1133) - https://phabricator.wikimedia.org/T244209 (10Andrew) [x] email sent to wikitech-l and cloud-announce [15:04:09] 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Trizek-WMF) > @Trizek-WMF I have CC'ed you here as last time you posted a message on technews for etherpad (T231403#5464235) - let's wait for a concrete day/time. You just need to... [15:26:47] o/ [15:27:03] marostegui: I believe you are around :D [15:27:32] addshore: I am about to logoff, jetlagg is hitting hard [15:27:36] do you need anything urgent? [15:28:04] * addshore was going to request another transaction dump similar to last time [15:28:16] addshore: sure, what do you need? [15:28:30] 2nd feb, around 11:44 [15:28:39] let me find some sort of ID [15:28:53] addshore: hehe you'll need to remind me the table and yeah, an ID would be nice :) [15:31:17] Yes, so the tables are those prefixed with wbt_* on wikidatawiki [15:31:27] 2nd feb 11:45 [15:31:57] would it be possible to get a dump of a bunch of transactions after a specific point in time? [15:32:03] yep [15:32:12] ie, after revision 1103620837 was created [15:32:25] so that was created 2nd feb 11:45? [15:32:28] sorry, revision 1108479589 [15:32:35] yes [15:32:45] 2nd feb, revision 1108479589 insertion and onwards [15:32:52] ok, checking wikidatamaster [15:33:04] I can take over and let you have a break if you want [15:33:35] I got I think :) [15:33:36] (I saw scary tickets on phab) [15:33:39] I mean, I just found it [15:33:44] cool [15:33:53] just letting you know a was back from lunch [15:33:53] everythin up to revision 1108479699 would be great [15:34:06] addshore: but what do you need exactly? [15:34:55] Well, I need the transactions for the secondary data updates that were done with edits 1108479589 and 1108479601 [15:35:05] *revisions [15:35:27] I am asking because the first entry I see for 1108479589 is: INSERT /* CommentStore::insertInternal */ INTO `revision_comment_temp` (revcomment_rev,revcomment_comment_id) VALUES (1108479589,xxxxx) [15:35:34] so not sure if that sort of thing is useful [15:35:35] if that itself is easy to grab out, otherwise im more than happy to sift through things myself [15:35:53] there a re bunch of other inserts on page, revision_actor etc [15:36:01] wb_changes etc [15:36:05] so, that line in itself is probably not useful. only things that interact with wbt_*tables will end up being useful [15:36:06] so not sure if all those are useful or not [15:36:15] ok, let me grab all the stuff for that id [15:36:56] assuming the size is not too large, I suggest to dumpt everything and he can later grep [15:37:09] as long at they don't get out of the production cluster [15:37:10] there are not many lines [15:37:27] addshore: I am going to leave the entries for those two IDs at mwmaint1002 at your home, is that ok? [15:37:40] yup, thanks! [15:38:36] addshore: the times I am looking at are: 11:32:27 till 11:50:45 [15:38:46] 👍 [15:38:48] is that ok or you'd like more? [15:39:03] that is a plenty big enough time range! [15:39:13] ok, let me create the files [15:40:48] hopfully I have figuired out the right criteria and this helps me find out what is happening :D [15:41:43] addshore: are you familiar with mysqlbinlog tool? [15:42:37] nope, but i can learn! [15:43:12] it is ok if you are not, just trying to be more efficient as this is the 2nd time we had to comb the transaction log [15:43:37] addshore: 1108479601.txt and 1108479589.txt at are your home [15:43:41] I dont believe i have access to view any of this myself? [15:43:52] otherwise i would definitely be up for it! [15:43:59] marostegui: thanks! [15:44:12] addshore: no, but if we just copy things in the original format you will speed us next time :-D [15:44:38] I am not sure I want to copy full binlogs somewhere else [15:44:47] no, not the full ones! [15:45:17] but copying offsets may be more efficient next time [15:45:22] ah :) [15:45:36] anyway, a discussion to have if there is a 3rd time [15:45:40] hehe [15:45:42] hopefully not [15:45:44] I am going to vanish I think :) [15:45:49] see you [15:45:52] o/ [15:46:27] addshore: remember to remove or call us to remove this from your home as soon as you are finished [15:46:35] jynus: ack! [15:47:41] jynus: unfortunately it looks like those 2 transactions are a bit too specific and don't include the things I'm looking for :/ [15:48:15] so I'll delete them now [15:50:38] let me know what do you need instead [15:50:45] Better conditions are probably, 2nd feb 11:45, DatabasePropertyTermStore::acquireAndInsertTerms SELECT FOR UPDATE from wbt_property_terms where wbpt_property_id => 356 [15:50:59] there should be 2 transactions in that time frame that match that [15:51:10] * addshore loads the docs for mysqlbinlog [15:51:13] one sec [15:55:51] so I won't be able to give you SELECT's for update as those are technically reads [15:56:14] but I can see INSERTs from that method at that time [15:56:45] let me know what you need from that time [15:57:42] are there 2 transactions with inserts or deletes for table wbt_property_terms with wbpt_property_id = 356 ? [15:57:57] (sorry, thought selects for update might show too) [15:58:11] I should have just said inserts / deletes to begin with [15:59:32] see some references to P356, but not wbpt_property_id = 356 so far (not finished the minute yet) [15:59:48] ah, I got it [16:00:08] DELETE /* Wikibase\Lib\Store\Sql\Terms\DatabasePropertyTermStore::acquireAndInsertTerms */ FROM `wbt_property_terms` WHERE wbpt_property_id = 356 AND ...? [16:00:29] 2020-02-02 11:45:45 that's the one? [16:00:51] ^addshore [16:01:34] looks right! [16:01:50] I hope there are 2 transactions that do something like that very near each other [16:02:31] what kind of write operations are you looking for? [16:02:39] as in, which tables, etc [16:03:19] all tables wbt_* [16:03:22] ok [16:03:29] everything else can be ignored [16:03:32] as if I just get you the previous and the next one [16:03:42] they are unlikely to be related, due to concurrency [16:03:50] indeed [16:04:02] the other transaction should also have a delete for wbpt_property_id = 356 [16:04:37] wbt_text? wbt_text_in_lang? [16:05:12] from Wikibase\Lib\Store\Sql\Terms\DatabaseTermIdsCleaner::cleanTextIds ? [16:07:40] yup, they all look good [16:07:45] I see 2 deletions with different wbpt_term_in_lang_id [16:07:55] I am guessing also insertions [16:08:00] yup [16:08:08] so that's it? [16:08:31] the important bit is the wbpt_property_id = 356 from the acquireAndInsertTerms method, then just everything else wbt_* table related in those transactions [16:08:56] and there should be 2 transactions (from 2 different edits) [16:09:08] yeah [16:09:54] woo! [16:16:15] I have it on mwmaint1002, making sure permisions are non public and will put it on your home [16:17:52] wbpt_property_id_356.sql at your ~ [16:18:39] please have a quick check that it works for you [16:18:54] so I can move to other stuff [18:09:47] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Patch-For-Review, and 2 others: Add index to recentchanges.rc_source - https://phabricator.wikimedia.org/T244080 (10Krinkle) I defer to #Growth-Team about what would or wouldn't change about the columns these queries are considering, and what index that m... [18:24:37] 10DBA, 10Performance Issue, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10Ladsgroup) Having something like this would be extremely useful for me. I have been looking for something like this for... [18:30:55] ^reminder of this, obviously not in a hurry, addshore [18:31:10] Thanks! [18:31:15] yes, i got distracted by alarms :D [18:31:17] (my last comment, not the phab) [18:33:05] it looks like exactly what I need thanks!!! :) [18:35:04] cool, I am leaving, but contact me somewhere async if you need something else [20:01:20] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: es1019: reseat IPMI - https://phabricator.wikimedia.org/T243963 (10Cmjohnson) @jcrespo yes, this seems to be an issue and when it's down I will see if there are any f/w updates for it. Is this something you need right away? I can put it on the schedule for Thur... [20:46:08] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: es1019: reseat IPMI - https://phabricator.wikimedia.org/T243963 (10Marostegui) @cmjohnson - Thursday sounds good. I can leave the host depooled, downtimed and off, so you can tackle it in your afternoon. Just leave it powered on once you are done and I will take... [21:37:24] Puppet question for you, was this done on purpose for some reason or is a typo from when we've created the multiinstance core roles? [21:37:27] class role::mariadb::core_multiinstance { [21:37:29] system::role { 'mariadb::core': [21:37:32] in modules/role/manifests/mariadb/core_multiinstance.pp [22:23:49] also it seems that in the same profile we call class { 'mariadb::config' without passing the 'replication_role' param, that then defaults to 'standalone' instead of the usual 'master'/'slave'