[06:58:33] <wikibugs_>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3060297 (10Marostegui) So, db2034 finished and as per our discussion yesterday evening it is currently like this: ```   PRIMARY KEY (`rev_id`,`rev_user`),   KEY `rev_ti...
[07:33:05] <wikibugs_>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#3060322 (10Marostegui) a:03Cmjohnson Hey Chris!  Would you have time to replace this disk today? Thanks!
[07:35:29] <wikibugs_>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1060 - https://phabricator.wikimedia.org/T158193#3029364 (10Marostegui)
[07:43:39] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060335 (10Marostegui) s4 is done. As it had the primary key, this is what was run (on the master - db1040): ``` ./osc...
[08:20:35] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060372 (10Marostegui) s5 (dewiki and wikidatawiki) are done. dewiki only needed the column and the index as the PK wa...
[08:37:53] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060395 (10Marostegui) s6 (frwiki,jawiki and ruwiki) done, only needed the column and the key, the PK was already ther...
[08:39:57] <jynus>	 Can I add T155605 to the deployments page?
[08:39:59] <stashbot>	 T155605: Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605
[08:40:21] <marostegui>	 jynus: I didn't add it because it wasn't a long running change
[09:06:42] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060415 (10Marostegui) s1 (enwiki) done, only needed the column and the index:  ``` ./osc_host.sh --host=db1052.eqiad....
[09:11:49] <jynus>	 s5 problems on dbstore2001?
[09:13:02] <marostegui>	 checking
[09:13:13] <jynus>	 A delete with type ALL
[09:13:20] <jynus>	 probably a missing index
[09:13:22] <marostegui>	 uh??
[09:13:35] <marostegui>	 oh i see
[09:15:01] <jynus>	 no, same schema
[09:17:02] <jynus>	 but for some reason it is not using wb_changes_change_time
[09:17:09] <jynus>	 on other slaves it does
[09:19:31] <marostegui>	 yes, I see no reason for it, as it is a pretty simple delete
[09:19:44] <marostegui>	 simple as in it is just using one index (or it should)
[09:20:21] <jynus>	 I would run analyze when the query finishes
[09:21:56] <marostegui>	 yeah, let's do that
[09:22:11] <jynus>	 actually, it is not one query
[09:22:13] <jynus>	 but many
[09:22:17] <jynus>	 I am going to stop the slave
[09:22:28] <jynus>	 even if that means contention for a few minutes
[09:22:33] <marostegui>	 all deletes actually
[09:22:44] <jynus>	 "stop the slave"
[09:26:58] <jynus>	 fixed, I think
[09:27:23] <jynus>	 it will take a few minutes for s5 to catch up
[09:27:48] <marostegui>	 it looks good now yes :)
[09:27:49] <marostegui>	 thanks
[09:27:57] <jynus>	 analyze worked
[09:28:07] <jynus>	 for some reason it was reejecting the index
[09:28:43] <jynus>	 hey, not all bugs are that bad, that require months of debugging!
[09:29:04] <marostegui>	 hahaha
[09:29:26] <jynus>	 and it is much easier than with tokudb!
[09:30:00] <marostegui>	 yeah, I still remember: https://jira.mariadb.org/browse/MDEV-10796
[09:43:10] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060532 (10Marostegui) s7 done for all wikis (centralauth doesn't have that table). Only the column and index was need...
[10:16:16] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060614 (10Marostegui) s2 is done for all wikis except l10nwiki, which doesn't have that table. The PK was there in al...
[10:20:47] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060617 (10TTO) >>! In T155605#3060614, @Marostegui wrote: > s2 is done for all wikis except l10nwiki, which doesn't h...
[10:21:45] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060620 (10Marostegui) >>! In T155605#3060617, @TTO wrote: >>>! In T155605#3060614, @Marostegui wrote: >> s2 is done f...
[10:55:24] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060718 (10Marostegui) So, to sum up:  s1 done s2 done s3 pending s4 done s5 done s6 done s7 done  I will start with s...
[11:17:20] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060763 (10Marostegui) For s3 I am doing batches of wikis, starting with those which don't have the PK (129 wikis in t...
[12:44:12] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060871 (10Marostegui) All the 129 wikis in s3 that didn't have the PK have been modified, so they now have the new PK...
[12:50:13] <milimetric>	 hey yall, when I see old_text = DB://cluster24/6158167 in the text table, where can I get that text from?
[12:52:07] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060901 (10Marostegui) I am also filtering the wikis that do not have that table in s3:  ``` zh_twwiki|zh_cnwiki|wikim...
[12:52:09] <milimetric>	 marostegui ^?
[12:52:28] <marostegui>	 milimetric: hi! where do you see that? 
[12:53:14] <milimetric>	 I got a rev_text_id from revision and did "select * from text where old_id = 16291975;"
[12:55:02] <milimetric>	 marostegui: ^
[12:55:10] <marostegui>	 and you get: DB://cluster24/6158167 ?
[12:55:29] <milimetric>	 yes, it's saying the revision text is stored externally
[12:55:39] <milimetric>	 but I didn't know where that is
[12:55:48] <Reedy>	 https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php
[12:56:09] <Reedy>	 look for 'cluster24' => [
[12:56:12] <marostegui>	 milimetric: yes, that is stored on the es servers
[12:56:21] <milimetric>	 oh wow, what are es servers?
[12:56:45] <marostegui>	 look what Reedy said :-)
[12:56:50] <milimetric>	 also, Reedy, you're an encyclopedia man
[12:56:53] <marostegui>	 haha
[12:56:53] <milimetric>	 thank you
[12:57:06] <Reedy>	 It's for all of the blobs
[12:57:16] <Reedy>	 well, most
[12:57:17] <milimetric>	 right, but where/how do I get to them?
[12:57:25] <Reedy>	 Depends how you're trying
[12:57:30] <Reedy>	 MW itself will do it all nicely
[12:57:37] <milimetric>	 oh, right, so there's a problem :)
[12:57:40] <milimetric>	 somewhat self-inflicted
[12:57:42] <Reedy>	 If you're doing the sql queries yourself... You need to connect to one of the hosts yourself
[12:57:55] <Reedy>	 You could poll the MW API or similar
[12:57:59] <milimetric>	 I deployed a new extension that hid some pages from MW
[12:58:31] <milimetric>	 they used to be on Config:Something and we configured Config:Dashiki: to be a sub-namespace of Config:, so now all those pages aren't visible anymore
[12:58:38] <milimetric>	 so the API can't get to them
[12:58:43] <Reedy>	 milimetric: namespaceDupes ?
[12:59:17] <milimetric>	 I heard there was a script yeah that moved stuff, is that what you mean?  I didn't want to run it because I didn't want to migrate all of them
[12:59:48] <milimetric>	 but when I tried this on beta, it didn't have this effect of hiding the page
[12:59:58] <Reedy>	 https://phabricator.wikimedia.org/P4999
[13:00:15] <Reedy>	 There's various config pages that look broken, but no Config:Dashiki
[13:00:40] <milimetric>	 right, Config:Dashiki would be the destination
[13:01:05] <Reedy>	 For all of those?
[13:01:16] <milimetric>	 well some I'd want to just delete
[13:01:18] <Reedy>	 Does MW actually support subnamespaces? Or does it just go lolno
[13:01:59] <milimetric>	 I don't know the details of that, Yuri's extension JsonConfig allows you to configure "sub-namespaces" but I don't know if they're an actual first-class concept or like made up
[13:02:22] <milimetric>	 I guess they're made up 'cause they're not in the db or anything
[13:02:25] <Reedy>	 You're having to ask? :P
[13:03:12] <milimetric>	 I'm embarassingly new to mediawiki extensions and how all this functions
[13:04:12] <milimetric>	 ok, so is there a way to pull text out of cluster24?
[13:04:36] <milimetric>	 I was just going to manually make the 6 configs I need to migrate and deal with the broken pages later
[13:06:34] <milimetric>	 we're getting what we need out of dumps right now, just wondering if there's another way
[13:06:35] <Reedy>	 milimetric: There's a load of pages in NS 0... With a Dashiki prefix
[13:06:57] <milimetric>	 yes, those are going to be deleted too, after I fix all this
[13:07:52] <Reedy>	 As for your actual question
[13:07:52] <Reedy>	 sql metawiki -h 10.64.16.186
[13:08:00] <Reedy>	 Should only be one table, blobs_cluster24
[13:08:28] <Reedy>	 so then select blob_text from blogs_cluster24 where blob_id = 6158167
[13:08:42] <milimetric>	 from tin?
[13:08:56] <Reedy>	 well, less blogs, more blobs
[13:09:05] <milimetric>	 yep, tin, thx!
[13:09:06] <Reedy>	 yup
[13:09:17] <Reedy>	 Of course
[13:09:22] <Reedy>	 That may or may not actually help you
[13:09:31] <Reedy>	 Cause it's compressed
[13:09:42] <milimetric>	 yep, that's ok, it's good to know how
[13:09:48] <milimetric>	 I need this for a lot of upcoming work
[13:10:49] <milimetric>	 so after I migrate, is the right way to fix all the old pages that namespaceDupes script?
[13:11:11] <Reedy>	 Depends what you mean by "migrate" :)
[13:11:18] <Reedy>	 I fixed all the Config: ones that were in NS 0
[13:11:40] <Reedy>	 So you should be able to interact with them more easily onwiki now
[13:11:57] <Reedy>	 Or do you mean the Dashiki prefixed ones?
[13:12:21] <Reedy>	 In that case... If you use MW to move them to Config:Dashiki, MW will automatically put them in the correct NS in the database
[13:13:53] <milimetric>	 Reedy: ok, I'll try that, thanks for fixing them!
[13:17:46] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2948316 (10Reedy) >>! In T155605#3060901, @Marostegui wrote: > I am also filtering the wikis that do not have that tab...
[13:17:57] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3060930 (10Marostegui) We have 753 wikis (yay) to alter that already have the PK on s3. I will do that in batches. The...
[13:20:20] <milimetric>	 Reedy: ok, so I tried to move just one of them and it looks like it doesn't account for the new content model of that namespace: https://meta.wikimedia.org/w/index.php?title=Config:Dashiki:LanguageReportcard&action=edit 
[13:21:01] <milimetric>	 so it's ok, I'll just move them manually by creating the new pages and flagging the old ones for deletion
[13:21:07] <Reedy>	 https://meta.wikimedia.org/wiki/Config:Dashiki:LanguageReportcard
[13:21:17] <Reedy>	 Should be fixable using https://meta.wikimedia.org/wiki/Special:ChangeContentModel
[13:21:24] <Reedy>	 I guess it was never set
[13:21:29] <Reedy>	 Let me login with my WMF account
[13:21:40] <milimetric>	 oh ok, can I get access to that?
[13:21:53] <milimetric>	 guess I'd have to be an Administrator
[13:21:55] <Reedy>	 Do you have a WMF account with staff/sysop or similar rights on it?
[13:22:05] <milimetric>	 not on meta, just on mediawiki
[13:22:20] <milimetric>	 Reedy: but don't fix it, there will be a bunch of others if I move them
[13:22:22] <Reedy>	 I was meaning global rights :)
[13:22:46] <Reedy>	 Just seems silly moving them out of the way, to delete them, to recreate them when you should be able to fix it
[13:23:26] <milimetric>	 I agree, but is it ok to put me in Administrators?  I just don't want to cause you more work
[13:24:03] <Reedy>	 Well, that's special
[13:24:07] <Reedy>	 "The action you have requested is limited to users in one of the groups: Administrators, WMF Support and Safety."
[13:24:10] <Reedy>	 Why can't my account do it
[13:24:19] <milimetric>	 oh yeah, I get that too
[13:24:27] <milimetric>	 if you can't do it... it means it's broken :)
[13:24:36] <Reedy>	 Well, that's with my "staff" account
[13:24:46] <Reedy>	 What's your username on meta?
[13:25:11] <milimetric>	 Milimetric (WMF)
[13:26:34] <Reedy>	 milimetric: done
[13:26:37] <Reedy>	 Granted myself the rights to check
[13:26:46] <Reedy>	 MW doesn't know it's another content...
[13:26:56] <milimetric>	 thanks Reedy :)
[13:27:01] <Reedy>	 So setting them to JSON you should be good to go
[13:27:17] <jynus>	 you should never touch the external storage directly
[13:27:36] <jynus>	 we use maintenance scripts for that
[13:27:45] <milimetric>	 nice, works well
[13:27:56] <milimetric>	 oh, I was never going to update jynus, just read the text
[13:28:05] <milimetric>	 but Reedy fixed it anyway so I can do everything I need on MW
[13:28:22] <milimetric>	 thanks very much all, this has been 1. enlightening 2. useful and 3. fun
[13:28:23] <milimetric>	 <3
[13:28:26] <jynus>	 prefer mediawiki wrappers, there is lot of old stuff to do things directly
[13:28:45] <Reedy>	 Hence my suggestion to use the MW API to query it :)
[13:28:53] <jynus>	 yes
[13:29:55] <jynus>	 even when there is some mixup, it is always to do the least amount of stuff manually
[13:30:11] <jynus>	 *better
[13:35:59] <milimetric>	 hm, Reedy the JSON content model works ok, but it's not the same as the one for newly created pages in that namespace, check out the difference between https://meta.wikimedia.org/wiki/Config:Dashiki:MultimediaHealth and https://meta.wikimedia.org/wiki/Config:Dashiki:Sample/tabs
[13:36:34] <Reedy>	 https://meta.wikimedia.org/w/index.php?title=Config:Dashiki:MultimediaHealth&action=edit
[13:36:38] <Reedy>	 The content format application/json+pretty is not supported by the content model JSON.
[13:36:49] <Reedy>	 The content format application/json+pretty is not supported by the content model JSON.
[13:37:02] <milimetric>	 right, so it's still broken I guess
[13:37:13] <Reedy>	 Sounds like it
[13:37:26] <Reedy>	 Presumably, Config or something isn't setting it correctly on creation?
[13:37:42] <Reedy>	 Though..
[13:37:46] <Reedy>	 You've changed that one
[13:37:52] <milimetric>	 right, kind of tangled at this point
[13:41:07] <Reedy>	 | page_id  | page_namespace | page_title                  | page_content_model |
[13:41:07] <Reedy>	 +----------+----------------+-----------------------------+--------------------+
[13:41:08] <Reedy>	 |  9959595 |            482 | Dashiki:FlowByWiki          | json               |
[13:41:08] <Reedy>	 |  8387927 |            482 | Dashiki:LanguageReportcard  | json               |
[13:41:08] <Reedy>	 |  8551857 |            482 | Dashiki:MultimediaHealth    | json               |
[13:41:11] <Reedy>	 | 10203443 |            482 | Dashiki:Sample/tabs         | JsonConfig.Dashiki |
[13:41:20] <milimetric>	 yes, was just looking at that
[13:41:39] <milimetric>	 so Special:ChangeContentModel won't let me do a custom content_model
[13:41:43] <Reedy>	 Should the extension be registering that contentmodel somewhere?
[13:42:30] <milimetric>	 It configures it but I guess I didn't know about Special:ChangeContentModel or how to register a model for it
[13:42:37] <Reedy>	 ContentHandlerDefaultModelFor and/or ContentHandlerForModelID
[13:43:24] <Reedy>	 Oh.. Is this JsonConfigModels
[13:43:34] <Reedy>	 Which we had fun and games with Zero extensions for
[13:43:44] <jynus>	 Is this related to https://phabricator.wikimedia.org/T125885 ?
[13:43:46] <milimetric>	 sounds right, but again, really out of my depth, just used the JsonConfig extension and then Yuri got let go so I got lost...
[13:43:56] <milimetric>	 no jynus 
[13:44:06] <jynus>	 not related to flow
[13:44:10] <jynus>	 I mean the same issue
[13:44:41] <milimetric>	 oh, uh doesn't sound like it
[13:45:16] <Reedy>	 milimetric: I think I can make a patch that should fix it
[13:45:29] <milimetric>	 Reedy: what are you thinking?
[13:46:24] <Reedy>	 	"JsonConfigModels": {
[13:46:25] <Reedy>	 		"Dashiki": {
[13:46:25] <Reedy>	 			"view": "Dashiki\\DashikiView"
[13:46:25] <Reedy>	 		}
[13:46:25] <Reedy>	 	},
[13:46:29] <Reedy>	 I think something like that
[13:46:48] <milimetric>	 would this have to be deployed to meta somewhere?
[13:46:52] <Reedy>	 Based on the fun and games we had with Zero and extension registration
[13:47:00] <Reedy>	 Needs to go into gerrit, merged/cherry picked, deployed from tin
[13:47:05] <Reedy>	 Easy enough to do
[13:47:50] <milimetric>	 so I need to fix these dashboards, and I don't want to put you on the hook to do an ad-hoc deploy because I screwed up
[13:47:59] <milimetric>	 so I can just move / create / delete my way out of this just fine
[13:48:35] <milimetric>	 up to you, really
[13:48:51] <Reedy>	 Might aswell get it fixed properly while it's fresh
[13:48:55] <Reedy>	 https://gerrit.wikimedia.org/r/340310
[13:49:56] <Reedy>	 milimetric: Even better, it's the SWAT window in 10 :)
[13:50:05] <milimetric>	 ok, merged
[13:50:22] <Reedy>	 Does jenkins not run on that repo? You shouldn't V+2 generally ;)
[13:51:31] <milimetric>	 oops, it runs, my bad
[13:51:41] <Reedy>	 heh
[13:51:53] <Reedy>	 I have a good habit of committing invalid json
[13:52:08] <milimetric>	 I checked it first (man, with morning brain I do everything manually)
[13:52:23] <milimetric>	 sorry I woke up to this little outage, not at my best
[14:09:30] <milimetric>	 Reedy: so now that's deployed but what does it all mean? :)
[14:10:05] <milimetric>	 should I move the pages and move them back again?
[14:12:26] <Reedy>	 I just noticed in your README, you suggest to do that in config
[14:13:46] <Reedy>	 I wonder if JsonConfig really isn't setup for helping to change the config of a page properly
[14:14:09] <milimetric>	 the config has... https://github.com/wikimedia/operations-mediawiki-config/blob/9d42afdff9ff9ba9c89ba39ddc9224d89c14e9fe/wmf-config/CommonSettings.php#L3185-L3199
[14:14:30] <milimetric>	 yeah, it's possible, without yurik I'm not sure
[14:16:52] <Reedy>	 milimetric: If in doubt, just fix it in the DB
[14:16:52] <Reedy>	 update page set page_content_model = 'JsonConfig.Dashiki' where page_namespace = 482 AND page_title LIKE 'Dashiki%';
[14:17:06] <Reedy>	 https://meta.wikimedia.org/wiki/Special:ChangeContentModel?pagetitle=Config%3ADashiki%3AMultimediaHealth
[14:17:10] <Reedy>	 Wrong link
[14:17:14] <Reedy>	 https://meta.wikimedia.org/w/index.php?title=Config:Dashiki:MultimediaHealth&action=edit
[14:17:16] <milimetric>	 oooh, I know someone whose name starts with a j that might be mad at us :)
[14:17:33] <Reedy>	 milimetric: Have a look at how the maintenance script in core does it :)
[14:17:45] <Reedy>	 				$dbw->update(
[14:17:45] <Reedy>	 					'page',
[14:17:45] <Reedy>	 					[ 'page_content_model' => CONTENT_MODEL_JSON ],
[14:17:45] <Reedy>	 					[ 'page_id' => $row->page_id ],
[14:17:45] <Reedy>	 					__METHOD__
[14:17:46] <Reedy>	 				);
[14:17:48] <milimetric>	 Reedy: so I still have to move a bunch of other pages, should I just do that and then run the update again?
[14:17:51] <milimetric>	 :)
[14:17:56] <Reedy>	 Yeah, that's what I'd do
[14:18:09] <Reedy>	 I'm gonna file a bug about not being able to change the content model to these JsonConfig "sub types"
[14:18:14] <Reedy>	 But I doubt anyone is likely to fix it
[14:18:15] <milimetric>	 ok, but I don't think I have rights to run that update, will you be around for a little longer?
[14:18:27] <milimetric>	 yeah, I was going to say, I'd file a task but who would work on it
[14:18:47] <Reedy>	 I need to go out to the pharmacy, but won't be too long so can always run it in a bit for you
[14:18:52] <milimetric>	 oh! never mind, I can just create / delete instead of move for the rest
[14:19:21] <milimetric>	 unless you object
[14:19:40] <Reedy>	 Well, are all those in hte Config namespace yours?
[14:20:09] <milimetric>	 everything in Config: I think is mine, and anything in Dashiki: is mine
[14:20:34] <milimetric>	 but a lot of that stuff just has to get deleted anyway
[14:20:46] <milimetric>	 yeah man, don't worry, I got this, thanks very much for the help, I know a lot more
[14:20:56] <Reedy>	 I was just going to suggest we can just change all of the config ns stuff to the dashiki content model
[14:21:25] <milimetric>	 nah, it's ok, more than half has to be deleted anyway
[14:21:54] <Reedy>	 heh
[14:22:52] <milimetric>	 and it's not like history for these things matters much, thanks again, really
[14:54:55] <moritzm>	 jynus, marostegui: any objections against upgrading apache now on dbmonitor*? (sec update)
[14:55:15] <jynus>	 no, go on
[14:55:19] <moritzm>	 ok
[14:55:51] <marostegui>	 :)
[14:56:08] <jynus>	 can you stop it completely for a second?
[14:56:15] <marostegui>	 me?
[14:56:25] <jynus>	 ^moritzm
[14:56:28] <marostegui>	 ah
[14:56:59] <jynus>	 (after upgrade, I mean)
[14:57:07] <jynus>	 there is something I want to check
[14:57:32] <moritzm>	 upgraded, shall I stop it now on 1001?
[14:57:37] <jynus>	 yes, please
[14:57:39] <jynus>	 ping me when done
[14:57:52] <moritzm>	 stopped
[14:58:04] <jynus>	 yeah, it is not in production
[14:58:07] <jynus>	 you can start it
[14:58:09] <moritzm>	 ok
[14:58:27] <jynus>	 I could check the config, but this is easier :-)
[14:58:37] <jynus>	 and more reliable
[14:58:56] <moritzm>	 :-)
[14:58:58] <jynus>	 we should move ther, soon
[15:25:08] <jynus>	 we have a few "Table definition has changed, please retry transaction (10.64.16.191)"
[15:25:21] <jynus>	 about user_groups
[15:26:02] <marostegui>	 where?
[15:26:37] <jynus>	 on the mw logs
[15:26:40] <marostegui>	 maybe that was a race condition?
[15:26:44] <marostegui>	 when the alter was going thru?
[15:27:00] <marostegui>	 (it is still ongoing)
[15:27:31] <marostegui>	 (only pending s3)
[15:27:54] <marostegui>	 I can only see db1077 (s3) and not enwiki which was altered in the morning, so maybe it was just when the alter was being done
[15:28:36] <jynus>	 it seems only 35 in the last 24 hours
[15:29:11] <marostegui>	 yeah and the last ones only s3 
[15:29:27] <jynus>	 I would note it in the ticket for the owner to have it a look, in case it was not just temporary
[15:29:36] <marostegui>	 good point
[15:29:39] <marostegui>	 I will do that now
[15:29:58] <jynus>	 better overreport than underreport
[15:30:54] <jynus>	 I was looking closely at the logs, because I am agraid my changes could also break stuff
[15:30:59] <jynus>	 *afraid
[15:31:17] <jynus>	 but the rate is so small that it is difficult to say
[15:40:09] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3061407 (10Marostegui) From those 753 pending wikis, 308 have already been altered in different batches. We have seen...
[16:15:49] <wikibugs_>	 10DBA, 10MediaWiki-Categories, 03Community-Tech-Sprint, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3061473 (10thiemowmde) > What are your thoughts on my comments at T158724#3048124 and T158724#3054267?  As I said: just use a hash funct...
[16:18:56] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3061475 (10Marostegui) No errors in the last hour since I stopped the ALTERs - looks good.
[16:26:13] <wikibugs_>	 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3061502 (10JAllemandou) Hi @Marostegui We have questions around that task ass it prevents us to move forward in productionizing our data extraction pipeline (one major goal for us this quarter /...
[16:44:04] <wikibugs_>	 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3061581 (10Marostegui) >>! In T149418#3061502, @JAllemandou wrote: > Hi @Marostegui > We have questions around that task ass it prevents us to move forward in productionizing our data extraction...
[17:14:44] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3061679 (10Marostegui) The following wikis were checksummed in s2 during the day with no issues:  bgwiki - no differences bgwiktionary - no differences cswiki - no di...
[17:51:52] <wikibugs_>	 10DBA, 06Community-Tech, 10MediaWiki-Categories, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3061779 (10DannyH)
[19:10:34] <wikibugs_>	 10DBA, 06Community-Tech, 10MediaWiki-Categories, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3062053 (10kaldari) >If this is a relevant issue you must change the column type to integer. You can create a lookup table for all known collat...
[19:46:49] <RainbowSprinkles>	 ema: Question...I see this from time to time in logstash and wondering what it's indicative of...
[19:47:00] <RainbowSprinkles>	 Warning: Unable to record MySQL stats with: EXPLAIN /* CheckUser::doUserIPsRequest */ SELECT * FROM `cu_changes` WHERE cuc_ip_hex = '<foo>' AND (1 = 1)
[19:47:10] <RainbowSprinkles>	 (not always CheckUser, but it seems to be the most common offender)
[19:49:14] <RainbowSprinkles>	 Oh, hmm, _joe_...you filed https://github.com/facebook/hhvm/issues/3760 ~3y ago
[20:03:18] <wikibugs_>	 10DBA: Investigate db1047 replication lag - https://phabricator.wikimedia.org/T159266#3062225 (10jcrespo)
[20:45:11] <wikibugs_>	 10DBA, 06Community-Tech, 10MediaWiki-Categories, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3062467 (10Bawolff) > I'm not worried about collisions. I'm worried about bloating an already huge table with more data for no practical reason...
[22:35:35] <wikibugs_>	 10DBA: Investigate db1047 replication lag - https://phabricator.wikimedia.org/T159266#3062951 (10Marostegui) It caught up before I was able to check it. I saw it 13 minutes delayed but when I got it to the terminal it was already up to date. Did you see which thread was lagging? S1 or S2? I will investigate furt...