[00:24:45] Phab down? [00:25:03] Oh, right, Phab deployment. :-) [00:27:15] James_F: fixed [00:27:33] twentyafterfour: Yup, thanks! [00:27:34] the throttling problems should be resolved now (I hope) [00:34:03] twentyafterfour is there a way to get the user from request without calling internal phab functions? [00:34:28] paladox: not really [00:34:33] hmm [00:34:39] you have to match the user's session cookie [00:40:39] twentyafterfour we could get phabricator conduit to do it. [00:40:49] if we whitelist a url like the getwhoami [00:41:29] https://stackoverflow.com/questions/40692037/how-to-obtain-logged-in-user-information-from-outside-phabricator-domain [00:41:29] making a conduit request from inside the rate limiting code would be a really bad idea [00:41:46] that would double the request load on the server [00:42:38] hmm [00:43:01] i think the rate limiter needs to be pushed furthur into the core so that you can access stuff like usernames. [01:26:37] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ArticleCreationWorkflow, 10MediaWiki-extensions-General, 10Patch-For-Review: Create composer package that contains most of the MediaWiki extension phan config instead of copy/pasting it each time - https://phabricator.wikimedia.org/T186315 (10j... [01:27:03] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-General, 10Patch-For-Review: Create composer package that contains most of the MediaWiki extension phan config instead of copy/pasting it each time - https://phabricator.wikimedia.org/T186315 (10jmatazzoni) [02:54:00] paladox: the reason it doesn't have access to usernames is because it's running early in the request, before most of the code is loaded. The idea is to reduce load on the server by processing the request as quickly as possible. [02:54:23] but beyond that, some conduit requests and obviously logged out sessions don't have a username [02:54:50] and requests for static resources actually go through php as far as I can tell [02:58:50] hmm [02:58:55] yeh [03:44:56] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10matmarex) I hit this as well, and I wasn't even writing a comment or reloading a million tabs at the time. Just researching some tasks. [03:46:08] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Josve05a) I got this right now as well. I was creating a task with exception crash code from AWB. I then pressed the save/create task button, and I got this message. I then tried to go back to recover the text, but it was gone.... [04:35:45] 10Release-Engineering-Team, 10Wikipedia-Android-App-Backlog: Disable/remove Jenkins Jobs for Android app - https://phabricator.wikimedia.org/T198862 (10Fjalapeno) @hashar thanks, so basically what we need to do is modify the config file to remove the jobs we no longer need, correct? We will be able to monito... [05:38:47] 10Phabricator: Not filling in a Due Date, tag list of a task is indented (in search results, workboard columns, etc) - https://phabricator.wikimedia.org/T199141 (10Cirdan) [05:40:38] 10Phabricator: Not filling in a Due Date, tag list of a task is indented (in search results, workboard columns, etc) - https://phabricator.wikimedia.org/T199141 (10Cirdan) Sorry for the incomplete description. I've updated the task accordingly. Is there a way to fix this by editing the affected tasks? (btw I ju... [05:41:03] 10Phabricator: Not filling in a Due Date, tag list of a task is indented (in search results, workboard columns, etc) - https://phabricator.wikimedia.org/T199141 (10Cirdan) [06:33:24] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:08:04] PROBLEM - Puppet errors on deployment-webperf11 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:13:23] RECOVERY - Puppet errors on deployment-deploy01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:54] (03CR) 10Hashar: [C: 032] Run seccheck for TorBlock [integration/config] - 10https://gerrit.wikimedia.org/r/444626 (owner: 10Umherirrender) [07:20:06] (03CR) 10Hashar: [C: 032] Run phan and seccheck for NukeDPL [integration/config] - 10https://gerrit.wikimedia.org/r/444618 (owner: 10Umherirrender) [07:21:31] (03Merged) 10jenkins-bot: Run seccheck for TorBlock [integration/config] - 10https://gerrit.wikimedia.org/r/444626 (owner: 10Umherirrender) [07:21:34] (03Merged) 10jenkins-bot: Run phan and seccheck for NukeDPL [integration/config] - 10https://gerrit.wikimedia.org/r/444618 (owner: 10Umherirrender) [07:36:32] 10Phabricator, 10Operations: Getting 'TOO MANY REQUESTS' error - https://phabricator.wikimedia.org/T199184 (10MarcoAurelio) I guess this is implemented via some puppet config. Please revert if I am mistaken. [07:48:05] RECOVERY - Puppet errors on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [0.0] [08:03:04] 10Phabricator, 10Operations: Getting 'TOO MANY REQUESTS' error - https://phabricator.wikimedia.org/T199184 (10Jc86035) [08:03:07] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:03:31] 10Phabricator, 10Operations: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:05:10] 10Phabricator, 10Operations: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:06:13] 10Phabricator, 10Regression: Creating subtasks sets more restrictive edit permissions than parent task: Author cannot edit their subtask anymore - https://phabricator.wikimedia.org/T199122 (10mmodell) Why would "Task author, administrators, WMF-NDA" deny the author? [08:07:03] 10Phabricator, 10Operations: Rate-limit is too harsh and affects human users - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:09:49] 10Phabricator, 10Operations: Rate-limit is too harsh and affects human users - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:11:53] 10Phabricator, 10Operations: Rate-limit is too harsh and affects human users - https://phabricator.wikimedia.org/T198974 (10Jc86035) [08:11:56] 10Phabricator, 10Patch-For-Review: Exclude WMDE IP from rate limiting / throttling - https://phabricator.wikimedia.org/T198612 (10Jc86035) [08:15:36] 10Phabricator: Rate-limit is too harsh and affects human users - https://phabricator.wikimedia.org/T198974 (10Aklapper) [08:16:02] 10Phabricator: Getting 'TOO MANY REQUESTS' error - https://phabricator.wikimedia.org/T199184 (10Aklapper) [08:24:52] (03PS1) 10Hashar: Migrate Daddio skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444813 (https://phabricator.wikimedia.org/T183512) [08:25:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [08:25:27] (03CR) 10Hashar: [C: 032] Migrate Daddio skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444813 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:28:03] (03Merged) 10jenkins-bot: Migrate Daddio skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444813 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:36:48] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:44:53] 10Release-Engineering-Team, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Patch-For-Review: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10Tgr) Actually the [[https://grafana.wikimedia.org/dashboard/db/edit-stash?pane... [08:48:15] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:00:18] 10Release-Engineering-Team, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Patch-For-Review: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10jcrespo) > watch the save timing and edit stash dashboards proactively There... [09:13:47] 10Release-Engineering-Team (Kanban), 10Splash, 10Patch-For-Review: 'resources/screen.less : Stylesheets should not both specify "media" and contain @media - https://phabricator.wikimedia.org/T193402 (10hashar) a:03hashar [09:17:34] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T191056 (10aaron) [09:17:48] 10Release-Engineering-Team, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Patch-For-Review: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10aaron) 05Open>03Resolved [09:20:27] 10Phabricator, 10Regression: Creating subtasks sets more restrictive edit permissions than parent task: Others cannot edit such subtasks anymore - https://phabricator.wikimedia.org/T199122 (10Aklapper) [09:20:53] 10Phabricator, 10Regression: Creating subtasks sets more restrictive edit permissions than parent task: Others cannot edit such subtasks anymore - https://phabricator.wikimedia.org/T199122 (10Aklapper) Sorry, that was my fault getting it wrong. No author involved; corrected now. [09:22:34] 10Phabricator: Not filling in a Due Date, tag list of a task is indented (in search results, workboard columns, etc) - https://phabricator.wikimedia.org/T199141 (10Aklapper) > Is there a way to fix this by editing the affected tasks? Which actual problem does this create when it comes to performing work? [09:31:36] 10Release-Engineering-Team (Kanban), 10Other-skins, 10Patch-For-Review: [mediawiki/skins/Tempo] resources/tempo.css : Stylesheets should not both specify "media" and contain @media - https://phabricator.wikimedia.org/T193404 (10hashar) a:03hashar [09:40:03] hashar: what I did so far, reading this page :) https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys [09:41:08] my git is already configured (git config --global...) [09:41:20] I've created .netrc file [09:41:52] cloned mediawiki/tools/release [09:42:35] but I'm not sure if I even need that repo on my machine, looks like all commands should be executed at deployment.eqiad.wmnet? [09:45:01] zeljkof: yeah on the cluster directly [09:45:07] that is way faster this way [09:45:35] the script that creates branches does a bunch of round trip with gerrit, so if it is in the same DC as Gerrit, it makes it way faster [09:46:09] last time I did it, I opened up an etherpad and copy pasted the 12 steps in a doc [09:46:21] then checked them as I progressed [09:46:39] 12 steps! [09:46:46] hm, let me find them :) [09:46:52] 2.1 to 2.12 [09:46:56] based on the table of content [09:47:15] the thing is [09:47:29] until your run scap, nothing will happen [09:47:35] (until someone else run scap obviously hehe) [09:47:39] ah, now I see, the first step that I missed is that I need to ssh to deployment.eqiad.wmnet, then create .netrc and clone the repo there, right? [09:48:01] yup [09:48:08] I've created the file and cloned the repo on my machine :D [09:48:12] ok, will amend the docs [09:48:48] and make sure the .netrc is only readable by you ( chmod go-rwx ~/.netrc ) [09:49:24] hashar: uh oh, well, I would miss that step [09:49:49] so literally that? `chmod go-rwx ~/.netrc` [09:50:53] the file is on a public host [09:51:00] and it has some api credentials [09:51:08] so yeah it should not be readable by group / others :] [09:52:03] my unix-fu is not strong :/ [09:54:09] (03PS1) 10Hashar: Migrate Splash, Tempo, Truglass to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444831 (https://phabricator.wikimedia.org/T193402) [09:54:45] hashar: ok, so I'll update the docs making it clear that that step 0 is ssh to deployment, setting up git and creating .netrc and making it readable by you [09:54:47] sounds good? [09:54:55] +1 [09:55:09] (03CR) 10Hashar: [C: 032] Migrate Splash, Tempo, Truglass to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444831 (https://phabricator.wikimedia.org/T193402) (owner: 10Hashar) [09:55:55] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [09:57:03] step 0: play some rage against the maching [09:57:27] (03Merged) 10jenkins-bot: Migrate Splash, Tempo, Truglass to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444831 (https://phabricator.wikimedia.org/T193402) (owner: 10Hashar) [09:58:14] zeljkof: https://www.youtube.com/watch?v=DKL4X0PZz7M First Aid Kit - My Silver Lining [10:00:58] I need more distorted guitars for ultimate concentration :) [10:01:45] zeljkof: then you want trance music https://www.youtube.com/watch?v=zHdy7dFKobQ [10:01:58] well Goa Trance [10:02:07] a sub genre of Trance that only lasted for a few years [10:02:43] hashar: this is my poison https://en.wikipedia.org/wiki/Rage_Against_the_Machine [10:17:52] (03PS1) 10Hashar: Skip php70 tests for PhpTags [integration/config] - 10https://gerrit.wikimedia.org/r/444833 [10:18:48] 10Beta-Cluster-Infrastructure, 10Pywikibot-core, 10Patch-For-Review, 10Pywikibot-tests: TestSiteGenerators.test_allpages_langlinks_enabled test fails - https://phabricator.wikimedia.org/T199085 (10Xqt) Seems it is solved upstream. No further failure seen in tests. [10:20:45] (03CR) 10jerkins-bot: [V: 04-1] Skip php70 tests for PhpTags [integration/config] - 10https://gerrit.wikimedia.org/r/444833 (owner: 10Hashar) [10:21:08] (03PS2) 10Hashar: Skip php70 tests for PhpTags [integration/config] - 10https://gerrit.wikimedia.org/r/444833 [10:23:19] (03PS3) 10Hashar: Skip php70 tests for PhpTags [integration/config] - 10https://gerrit.wikimedia.org/r/444833 [10:23:53] hashar: ok, so I have done the new step 1, setup https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Setup [10:24:33] good [10:24:43] the next step is to run the script that asks Gerrit to create the branches [10:24:50] make sure to use the proper version number :] [10:25:43] hashar: ok, so this does not make much sense to me [10:25:44] https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_Gerrit [10:26:57] should I run `./make-wmf-branch...` or `php make-wmf-branch...`? or either would work? should we just pick one? `./make-wmf-branch...`? [10:31:28] hashar: ok, looks good? [10:31:38] should I use `screen`? [10:36:15] 10Release-Engineering-Team, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Patch-For-Review: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10daniel) > Seems to be back to pre-MCR levels (save timing in general too, as f... [10:36:54] (03PS4) 10Hashar: Skip php70 tests for PhpTags and migrate to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444833 (https://phabricator.wikimedia.org/T188585) [10:37:09] zeljkof: if you loose your ssh connection, yeah screen will be helpful :] [10:37:35] you can even record the terminal session with "script" [10:37:38] $ screen [10:37:46] hashar: ok, I'll add it to the docs [10:37:49] $ script deploy0710.log [10:37:51] :] [10:38:00] this way you can revisit what happened by browsing the log file [10:38:13] hashar: hm, would you recommend adding that to the docs? [10:38:29] not really [10:38:34] or just `screen (whatever)` [10:38:51] but it might be worth mentionning to use screen and script yes [10:38:56] I dont know. It is not that important [10:39:09] how long does it take to run the script? [10:39:15] seconds? minutes? hours? [10:39:25] no clue [10:39:47] when we did the hangout with Tyler the whole session took 1 hour [10:39:48] can I break stuff if I lose connection while it's running? :) [10:40:04] ok, so in that case screen it is :) [10:40:06] (03CR) 10Hashar: [C: 032] Skip php70 tests for PhpTags and migrate to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444833 (https://phabricator.wikimedia.org/T188585) (owner: 10Hashar) [10:40:22] ^ ^lets break CI :D [10:42:25] (03Merged) 10jenkins-bot: Skip php70 tests for PhpTags and migrate to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444833 (https://phabricator.wikimedia.org/T188585) (owner: 10Hashar) [10:46:56] hashar: ok, running this https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_Gerrit [10:47:07] it will take minutes, not hours, I guess :) [10:48:55] reports that the VisualEditor tag n phab gives you a 404 page when you go to look [10:49:05] https://phabricator.wikimedia.org/tag/visualeditor/ [10:49:15] the tag exists; you can see it on some tasks [10:49:20] 404 for me [10:49:52] https://phabricator.wikimedia.org/project/view/480/ examples here, 2nd, 3rd tasks in column [10:50:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:51:24] apergos: please feel a task against #Phabricator and #VisualEditor ? :] [10:51:37] I'll ask the reporters to do so, thanks! [10:51:46] I feel recursion in the air [11:03:45] hashar: ok, this finished https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_Gerrit [11:03:49] in about 15 minutes [11:04:45] hashar: can I do the next step? or, which step is the first step that should be done in the actual window [11:05:19] yup you can clone the new branch [11:05:23] scap prep [11:05:28] then apply the security patches [11:06:59] ok, swat is self-service today :) so I have the time [11:07:35] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:08:09] 10Beta-Cluster-Infrastructure, 10Pywikibot-core, 10Patch-For-Review, 10Pywikibot-tests: TestSiteGenerators.test_allpages_langlinks_enabled test fails - https://phabricator.wikimedia.org/T199085 (10Dalba) >>! In T199085#4411302, @Xqt wrote: > Seems it is solved upstream. No further failure seen in tests. I... [11:13:55] hashar: um, both Chad and Darian are no longer around, right? https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Apply_security_patches [11:14:14] yup [11:14:33] ok, so I'll remove them, but leave the channel name [11:14:50] or is there somebody else to contact? [11:15:12] zeljkof: probably reach out to the security team, like sam or bawolff [11:15:25] and see who they suggest listing [11:15:36] p858snake: is there a list to point to? [11:15:47] to avoid docs being out of date again in the future [11:17:34] probably https://wikimediafoundation.org/wiki/Staff_and_contractors#Security or https://www.mediawiki.org/wiki/Wikimedia_Security_Team [11:20:23] p858snake: thanks! [11:24:41] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Deskana) [11:29:23] Deskana: the only thing I can think of, is disabling the workboard and trying to turn it on again, but I can't remember if losses all the column linkage when you turn it back on [11:33:17] hashar: ok, applying security patches https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Apply_security_patches [11:35:34] I remember back in the day when that channel was a secret, and mentioning it was like, mentioning he who must not be named [11:36:04] well, it's on a public page... [11:36:21] these days it is pretty public knowledge :) [11:52:44] hashar: ok, I have updated the docs, I guess this is what I need to do https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Apply_security_patches [12:14:29] 10Release-Engineering-Team (Watching / External), 10Scap, 10Services (done): Allow deploying services from a branch - https://phabricator.wikimedia.org/T198958 (10Pchelolo) 05Open>03Invalid It indeed works, thank you and sorry for false report. [12:30:36] 10Beta-Cluster-Infrastructure, 10Pywikibot-core, 10Pywikibot-tests: TestSiteGenerators.test_allpages_langlinks_enabled test fails - https://phabricator.wikimedia.org/T199085 (10Xqt) a:05Xqt>03None [12:51:55] (03PS1) 10Hashar: Chrome: do not rate limit history.pushState() [integration/quibble] - 10https://gerrit.wikimedia.org/r/444871 (https://phabricator.wikimedia.org/T198171) [12:52:21] (03CR) 10Hashar: [C: 032] Chrome: do not rate limit history.pushState() [integration/quibble] - 10https://gerrit.wikimedia.org/r/444871 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [12:53:11] (03Merged) 10jenkins-bot: Chrome: do not rate limit history.pushState() [integration/quibble] - 10https://gerrit.wikimedia.org/r/444871 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [12:53:52] (03CR) 10jenkins-bot: Chrome: do not rate limit history.pushState() [integration/quibble] - 10https://gerrit.wikimedia.org/r/444871 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:02:24] (03PS1) 10Hashar: docker: quibble 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444876 (https://phabricator.wikimedia.org/T198171) [13:02:41] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444876 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:04:20] (03PS1) 10Hashar: Bump Quibble jobs based on Stretch to 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444877 (https://phabricator.wikimedia.org/T198171) [13:04:22] (03Merged) 10jenkins-bot: docker: quibble 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444876 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:05:29] !log Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 | T198171 [13:05:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:05:33] T198171: [RevisionSlider] QUnit test ext.RevisionSlider.DiffPage "Push state" fails due to Chrome rate limiting history.pushState - https://phabricator.wikimedia.org/T198171 [13:10:13] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Esanders) p:05Triage>03High [13:11:21] zeljkof: have you done the security patches yet ? :) [13:11:39] hashar: finishing the last one right now [13:13:57] hashar: security patches done [13:14:16] thcipriani: should I wait for you to finish the cleanup? [13:14:38] zeljkof: yes please, should only take a few minutes [13:15:52] no problemo, señor [13:34:40] (03CR) 10Hashar: [C: 032] Bump Quibble jobs based on Stretch to 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444877 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:37:06] (03Merged) 10jenkins-bot: Bump Quibble jobs based on Stretch to 0.0.20 [integration/config] - 10https://gerrit.wikimedia.org/r/444877 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:47:22] (03PS1) 10Hashar: Migrate RevisionSlider to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444884 (https://phabricator.wikimedia.org/T198171) [13:47:33] (03CR) 10Hashar: [C: 032] Migrate RevisionSlider to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444884 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [13:49:57] (03Merged) 10jenkins-bot: Migrate RevisionSlider to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444884 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [14:11:56] hashar: Have you seen https://phabricator.wikimedia.org/T189560 ? [14:12:04] It seems it installs the dependancies from composer... [14:12:22] But then they're not being loaded (or have been removed/wiped?) by the time the tests are being run [14:17:34] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: PHP Fatal Error: Class 'Wikimedia\EquivSet\EquivSet' not found - https://phabricator.wikimedia.org/T189560 (10Reedy) It's already in extension.json for AbuseFilter https://integration.wikimedia.org/ci/job/release-quibble-vendor-mysql-hhvm-doc... [14:43:02] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Aklapper) [14:55:30] PROBLEM - Puppet errors on deployment-elastic08 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:58:08] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Deskana) p:05High>03Unbreak! I was hoping this was a transient issue, but it's been going on for hours now and it's preventing the team from doing their work, so I'm escalating the priority. [15:15:18] 10Release-Engineering-Team, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Patch-For-Review: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10Imarlier) @daniel It may be possible to change the visualization in order to m... [15:35:06] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [15:55:38] Is https://doc.wikimedia.org not updating very often? I see stuff in core on master that's not there [15:56:13] What do you think is missing/out of date? [16:05:46] Reedy: no deprecation notice in WikiPage:doEditContent() [16:05:54] and PageUpdater isn't there [16:05:56] https://doc.wikimedia.org/mediawiki-core/master/php/classWikiPage.html#a78bb35e210e88e3dcc3e211e43995af0 [16:06:10] Generated on Thu May 24 2018 15:10:30 for MediaWiki by [16:07:01] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/991/console [16:07:11] 16:05:51 Error: You might be using an older PHP version (PHP 5.6.33-0+deb8u1). [16:07:11] 16:05:51 MediaWiki 1.32 needs PHP 7.0.0 or higher or HHVM version 3.18.5. [16:07:11] 16:05:51 [16:08:50] 10Continuous-Integration-Config: mediawiki-core-doxygen-docker failing due to PHP version errors - https://phabricator.wikimedia.org/T199243 (10Reedy) [16:09:17] 10Continuous-Integration-Config: mediawiki-core-doxygen-docker failing due to PHP version errors - https://phabricator.wikimedia.org/T199243 (10Reedy) [16:11:02] 10Continuous-Integration-Config: mediawiki-core-doxygen-docker failing due to PHP version errors - https://phabricator.wikimedia.org/T199243 (10Reedy) [16:11:27] Reedy: well, it says "master" in the URL [16:11:30] So something's off [16:11:51] Hence my pasting of "Generated on Thu May 24 2018 15:10:30 for MediaWiki by" [16:12:16] It's seemingly broken for REL1_31 [16:12:54] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/969/console [16:12:56] Broken for master [16:13:37] 10Continuous-Integration-Config: mediawiki-core-doxygen-docker failing due to PHP version errors - https://phabricator.wikimedia.org/T199243 (10Reedy) p:05Triage>03High [16:15:32] 10Continuous-Integration-Config: mediawiki-core-doxygen-docker failing due to PHP version errors - https://phabricator.wikimedia.org/T199243 (10Reedy) [16:15:44] Reedy: ah yeah cool I see the bug now...... thx!!!! [16:15:50] I mean the Fab bug report [16:16:03] * AndyRussG groggily runs for more coffeeee [16:16:07] Kinda sad no one has noticed in a few months [16:16:38] Maybe not a lot of people have a habit of googling "mediawiki SomeCoreClass class reference" [16:16:51] Reedy: for EquivSet ( https://phabricator.wikimedia.org/T189560 ) I might look at it tonight [16:17:03] Cheers :) [16:17:07] It's a weird one [16:17:11] Reedy: maybe it is simply not in vendor.git [16:17:20] It isn't in REL1_31 [16:17:23] And shouldn't be [16:17:26] Reedy: thx again :) [16:17:56] It is in master, but for release branches... it's not. AFAIK we don't bundle AF [16:19:03] Krenair: I finally got a chance to note the config info for the old dumps puppetmaster in deployment-prep, you can kill it. I should have everything in my notes now [16:19:05] thanks again [16:35:36] 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General, 10Services (designing): Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229 (10Mholloway) [16:51:43] 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Follow up with platform team regarding refactoring plans - https://phabricator.wikimedia.org/T189062 (10Jrbranaa) 05Open>03Resolved Moving to resolved. I've been engaging with Platform team as well as Search Platform team regarding refactoring as... [17:07:51] hey y'all... We still have that weird random selenium test failure that makes it virtually impossible to merge commits; I can't seem to find the actual test to see even who to contact about fixing it... I might not be reading things right? but can we consider deactivating it until it's fixed? :\ [17:07:54] see example: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6989/consoleFull [17:08:29] 1) Special:RecentChanges shows page creation: [17:08:29] 'BeforeEach-name-0.20204109014353766-Iñtërnâtiônàlizætiøn' === '0.2524668712533231-Iñtërnâtiônàlizætiøn' [17:10:21] thcipriani: I think you looked at it last week... do you happen to remember whose test this is? [17:12:14] hrm, I recall looking at this task https://phabricator.wikimedia.org/T198137 and I recall some talk of a .quibble file to skip tests [17:12:47] ok great I thought I remembered a task and for some reason couldn't find this one, was about to file another [17:14:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10Niharika) Can we please get #3 fix... [17:14:58] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10Mooeypoo) Another failure; same as... [17:16:59] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [17:19:02] 10Release-Engineering-Team: Investigate and propose record of origin (ROO) for deployed code (currently Developers/Maintainers page) - https://phabricator.wikimedia.org/T199253 (10Jrbranaa) [17:24:48] 10Release-Engineering-Team: TEC13:O1:O1.1:Q1 Goal - Investigate and propose record of origin (ROO) for deployed code (currently Developers/Maintainers page) - https://phabricator.wikimedia.org/T199253 (10Jrbranaa) [17:25:38] 10Release-Engineering-Team (Kanban), 10Analytics-Tech-community-metrics, 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891 (10Jrbranaa) [17:25:41] 10Release-Engineering-Team: TEC13:O1:O1.1:Q1 Goal - Investigate and propose record of origin (ROO) for deployed code (currently Developers/Maintainers page) - https://phabricator.wikimedia.org/T199253 (10Jrbranaa) [17:35:23] I'm now getting the "too many requests" thing from Phab at my house :( [17:35:54] RoanKattouw: Clearly you're working too much. [17:37:58] Right during our triage meeting too :( [17:42:39] 10Release-Engineering-Team: TEC13:O3.2:Q1 Goal - Platform and Search Platform teams are using Technical Debt Management PoC - https://phabricator.wikimedia.org/T199259 (10Jrbranaa) [17:46:00] 10Release-Engineering-Team, 10Code-Health: TEC13:O4.1:Q1 Goal - Define base Code Health metric set. - https://phabricator.wikimedia.org/T199261 (10Jrbranaa) [17:48:56] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10atgo) Hi @mmodell. Would it be possible for Due Dates and their changes to be logged in the revision history of the task? This came up recently with some due date vandalism, and would also be useful for visibility into the hi... [17:52:08] 10Release-Engineering-Team: TEC13:O3.4:Q1 Goal - Identify key Tech Debt areas (Platform) - https://phabricator.wikimedia.org/T199262 (10Jrbranaa) [18:02:41] (03PS1) 10MarcoAurelio: Edit Project Config [extensions/Transliterator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/444930 [18:03:03] (03PS2) 10MarcoAurelio: Edit Project Config [extensions/Transliterator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/444930 [18:03:12] (03CR) 10MarcoAurelio: [V: 032 C: 032] Edit Project Config [extensions/Transliterator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/444930 (owner: 10MarcoAurelio) [18:08:00] (03PS2) 10MarcoAurelio: Add Phan and sec-check tests for NukeDPL [integration/config] - 10https://gerrit.wikimedia.org/r/444444 [18:11:09] (03Abandoned) 10MarcoAurelio: Add Phan and sec-check tests for NukeDPL [integration/config] - 10https://gerrit.wikimedia.org/r/444444 (owner: 10MarcoAurelio) [18:16:18] 10Release-Engineering-Team (Kanban), 10User-greg, 10Wikimedia-extension-review-queue: Re-think/factor [[mw:Review queue]] and generally the process of getting new code into production - https://phabricator.wikimedia.org/T195244 (10Quiddity) [18:16:56] 10Release-Engineering-Team (Kanban), 10User-greg, 10Wikimedia-extension-review-queue: Re-think/factor [[mw:Review queue]] and generally the process of getting new code into production - https://phabricator.wikimedia.org/T195244 (10Quiddity) I've added some relevant (I think) links to the description, to aid... [18:26:58] (03PS2) 10Jforrester: [mwgate] Drop MwEmbedHandler, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443003 [18:27:00] (03PS2) 10Jforrester: [CirrusSearch] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443004 [18:27:02] (03PS2) 10Jforrester: [TimedMediaHandler] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443005 [18:27:04] (03PS3) 10Jforrester: [MwEmbedSupport] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/443006 (https://phabricator.wikimedia.org/T197918) [18:31:31] !log deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in T72792) [18:31:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:31:36] T72792: Set up puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792 [18:34:11] greg-g: There's a big problem with the VE workboard. It's giving a 404: https://phabricator.wikimedia.org/T199207 [18:34:22] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Set up puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792 (10Krenair) cleaned that up today per Krenair: I finally got a chance to note the config info for the old dumps puppetmaster in... [18:34:28] greg-g: Is this something you can take a look at soon? It's been this way for 12 hours or so. [18:34:39] did you disabled it? [18:34:54] Unbreak Now? [18:35:32] (03CR) 10Krinkle: [C: 032] "I'm not sure it will pass with quibble, but worth a try. The jobs are quite different. I suppose if not, we'll just proceed to -archived." [integration/config] - 10https://gerrit.wikimedia.org/r/443003 (owner: 10Jforrester) [18:36:01] (03CR) 10Krinkle: [C: 032] [CirrusSearch] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443004 (owner: 10Jforrester) [18:36:08] (03CR) 10Krinkle: [C: 032] [TimedMediaHandler] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443005 (owner: 10Jforrester) [18:36:14] Krinkle: Thanks! [18:36:21] (03CR) 10Krinkle: [C: 032] "Or that." [integration/config] - 10https://gerrit.wikimedia.org/r/443006 (https://phabricator.wikimedia.org/T197918) (owner: 10Jforrester) [18:36:34] yw :) [18:37:02] (03Merged) 10jenkins-bot: [mwgate] Drop MwEmbedHandler, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443003 (owner: 10Jforrester) [18:37:17] I'm looking at it right now and I don't see why this is happening. The hashtags are okay, the workboard is on... [18:37:31] Deskana: looking [18:37:48] twentyafterfour: I can't figure out why the VE workboard is 404ing... https://phabricator.wikimedia.org/project/board/483/ [18:38:13] (03Merged) 10jenkins-bot: [CirrusSearch] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443004 (owner: 10Jforrester) [18:38:15] (03Merged) 10jenkins-bot: [TimedMediaHandler] Stop injecting MwEmbedHandler as a dependency, it's a no-op [integration/config] - 10https://gerrit.wikimedia.org/r/443005 (owner: 10Jforrester) [18:38:18] (03Merged) 10jenkins-bot: [MwEmbedSupport] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/443006 (https://phabricator.wikimedia.org/T197918) (owner: 10Jforrester) [18:38:21] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Krenair) https://phabricator.wikimedia.org/project/board/483/manage/ shows it's enabled... let's try disabling and re-enabling to see if that helps things? [18:38:51] greg-g: weird? [18:39:17] twentyafterfour: disabling/re-enabling a workboard keeps the columns/where the tasks are, right? [18:39:51] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Krenair) hm nope, https://phabricator.wikimedia.org/project/board/483/disable/ gives a 404 when you press the button [18:40:12] Krenair: huh [18:40:15] greg-g: yes [18:40:32] Hauskatze: thanks, I thought so, just wanted to double check, would hate to have that turn out badly :) [18:41:04] (03CR) 10Krinkle: "This wasn't deployed but is going to be pushed now as part of me deploying https://gerrit.wikimedia.org/r/#/c/integration/config/+/443006/" [integration/config] - 10https://gerrit.wikimedia.org/r/444884 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [18:41:06] hasharDinner: ^ [18:41:12] Unless recent phabricator updates have changed that... [18:41:29] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10matmarex) You can still view the list of tasks using task search: https://phabricator.wikimedia.org/maniphest/query/9WFEhfANhE6N/#R (note that it is paginated). (Frankly, it seems more useful without th... [18:41:40] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884 as well) [18:41:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:42:00] greg-g: yeah [18:42:46] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10Krenair) I fiddled a bit, tried reverting the sprint change earlier, no luck: https://phabricator.wikimedia.org/project/manage/483/#47174 [18:45:17] 10Phabricator, 10VisualEditor: 404 on VisualEditor workboard - https://phabricator.wikimedia.org/T199207 (10MarcoAurelio) Same here. No luck. [18:45:53] https://phabricator.wikimedia.org/project/board/483/manage/ [18:47:12] hitting on "disable workboard" but nothing happens [18:48:54] Hauskatze, yeah look at the console [18:48:55] it's 404ing [18:49:54] I was going to make a joke about a PS4 console but no, bad idea :P (/me has no videoconsoles though) [18:50:29] twentyafterfour, idea [18:50:34] can you look up the project in the DB? [18:50:38] check the hasWorkboard field [18:51:20] Thanks for looking into it everyone! I'm heading off for the evening now. I'll catch up in the morning. [18:52:07] POST https://phabricator.wikimedia.org/project/board/483/disable/ 404 () [18:52:07] 20:50:34.025 core.pkg.js:220 XHR failed loading: POST "https://phabricator.wikimedia.org/project/board/483/disable/". [18:52:14] yeah [18:53:10] Krenair it works disabling it here https://phabricator.wikimedia.org/project/483/item/configure/global/ [18:53:39] greg-g: o/ i'd like to set up a new maps testing instance in the beta cluster for T172090 (equivalent to deployment-maps03 but for testing on stretch). is there any special ceremony around this for the beta cluster or should i just go for it? [18:53:40] T172090: move maps-test cluster to cloud infrastructure - https://phabricator.wikimedia.org/T172090 [18:53:41] that just hides the workboard, don't disable it, right? [18:54:05] * Hauskatze looks at Conduit [18:54:12] it says disable [18:54:20] yes [18:54:24] but it's not referring to the workboard itself [18:54:30] it's referring to the link to the workboard [18:54:31] I think [18:56:19] mdholloway: no special ceremony no :) [18:56:32] greg-g: great, thanks! [18:56:57] you'll find beta has a bunch of spare quota right now due to all the legacy instances I just got rid of. try not to use too much of it [18:57:02] https://phabricator.wikimedia.org/tag/visualeditor/ returns 404 now [18:57:07] oh bloody hell [18:57:10] uri": "https://phabricator.wikimedia.org/tag/visualeditor/", [18:57:14] paladox: yeah [18:57:22] seems we're looking at the same stuff [18:57:53] PHID-PROJ-dafezmpv6huxg3taml24 [18:58:03] https://phabricator.wikimedia.org/project/view/483/ [18:58:08] https://phabricator.wikimedia.org/project/profile/483/ is visible [18:59:13] Krenair: it looks like the default order for the workboard got corrupted [18:59:17] rather, deleted [18:59:31] mdholloway, also, try to name the instances consistently, ensure puppet runs successfully, etc. [19:00:11] Krenair: will do [19:00:13] Hauskatze, paladox: alright it looks like this is because someone made the workboard the default view thing for the project [19:00:23] oh [19:00:31] that caused the corruption? [19:00:39] I've set it back and https://phabricator.wikimedia.org/tag/visualeditor/ works again [19:00:51] Of course, phabricator doesn't log this against the project [19:01:08] Krenair: wtf? [19:01:12] paladox, no, that just caused /tag/visualeditor/ to break [19:01:18] oh [19:01:46] yep, but the workboard still 404s [19:01:52] yeah [19:02:02] twentyafterfour, what bit are you wtfing about/ [19:02:02] ? [19:02:10] if it's all the bits that's ok too [19:02:15] this is phabricator [19:02:22] Oh Father Phabricator, make us privy to your misteries... [19:02:46] Krenair: sorry I thought you were saying you fixed it by changing the default menu item [19:03:05] twentyafterfour, I fixed a different problem that someone added to the mix while everyone else was trying to figure out the original problem [19:03:23] np [19:05:29] twentyafterfour, surely if the workboard's order gets corrupted you get errors when trying to view the workboard, instead of it just pretending it doesn't exist? [19:07:00] Krenair: https://github.com/phacility/phabricator/blob/master/src/applications/project/controller/PhabricatorProjectBoardViewController.php#L84 [19:07:19] it surely does 404 if the saved order record is missing [19:07:26] which shouldn't happen [19:07:42] :S [19:08:01] okay shouldn't it be throwing exceptions or at least logging something there [19:09:34] Krenair: yeah for sure... [19:09:56] The other question is "what changed since last night?". [19:10:08] well someone did edit the project [19:10:25] though I don't fully understand the implications of it [19:10:31] https://phabricator.wikimedia.org/project/manage/483/#47174 [19:10:42] Oh, hmm. [19:10:46] Krenair that was today [19:10:51] and after the task was filled [19:10:54] Yeah, I left that cruft alone in case it broke things. Clearly I was right to do so. :-( [19:11:10] on the other hand, Deskana wrote at 18:34 UTC in this channel "It's been this way for 12 hours or so." [19:11:42] (Historically it had to be set to be a sprint to be managed by the extension that let us set points on its tasks.) [19:11:43] oh right, and it was after the task got filed. well spotted paladox [19:12:04] James_F, finding out what changed might be difficult [19:12:17] phabricator doesn't keep records of all things people change [19:12:31] There was an update to Phab to try to fix the IP DoS issue. [19:12:46] Did any other config changes go out at the same time? [19:15:42] https://phabricator.wikimedia.org/project/board/483/query/cXWUBtEYlPCo/ [19:16:02] "This workboard has been disabled, but can be restored to its former glory." [19:17:29] I don't see that text but I do see the workboard [19:17:40] Krenair: because I enabled the workboard already [19:17:47] yay [19:17:57] and now the only remaining thing is to fix the default filter query [19:18:10] I don't know what it's supposed to be set to though [19:18:19] https://phabricator.wikimedia.org/project/board/483/ still 404 for me twentyafterfour [19:19:39] in the meanwhile we can fix scap as well... https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/441571/ hint hint :P [19:21:00] I've set it to default to open tasks [19:21:15] since i'm pretty sure that's the normal one [19:22:10] yeah... that should be the behavior when that record can't be found. much more useful than a 404 [19:24:05] works again :D [19:25:55] Reedy: tldr mediawiki/vendor.git REL1_31 does not have "wikimedia/equivset": "1.3.0" [19:26:23] Reedy: you are 100% right that equivset does get installed, but that is a composer install && composer test being run in the extension directory [19:26:40] Reedy: for the sole purpose of running "composer test". The resulting installation is then entirely discarded [19:27:48] twentyafterfour, what was the trick to get around this, did you have to do anything special to get that cXWUBtEYlPCo ID for the URL? [19:28:25] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) @atgo: I'm actually not sure why the aren't logged. I'll see what I can do. [19:28:30] or was that just an ID from a different workboard's filtering? [19:29:42] Hey! Is there a process to grant projectadmin to deployment-prep in horizon? [19:30:00] in horizon? [19:30:02] you mean, in keystone? [19:30:13] Krenair: probably [19:30:35] * Krenair shrugs [19:30:45] ask nicely and ye shall receive [19:30:52] :) [19:31:00] let's see if I can remember how we do this [19:31:30] I'm already admin on deployment-prep, the question is should I just give rights to other people, or should that be tracked somewhere [19:31:40] oh right [19:31:50] gehel: if in doubt, create a task :] [19:32:14] well there's no NDA necessary [19:32:18] do we still use deployment-tin or ... ? [19:32:30] if they're a well-known wikimedian it should be fine [19:32:42] yeah... if I can do that in 2 minutes right now, I much prefer unblocking mdholloway right now than creating a task :) [19:32:44] Hauskatze, yeah [19:32:52] no strech for us yet, okay [19:32:54] mdholloway is fine to go straight in [19:33:05] ssh -a deployment-tin [19:33:14] ok, thanks for the help! [19:33:17] Hauskatze, -tin is a jessie host [19:33:27] you want -deploy01 [19:33:42] which should take over as the main deployment server at some point [19:34:25] deploy01 has a big warning saying not to use it [19:34:35] so I'll use tin for now [19:35:07] yeah same for -mira [19:35:13] yup [19:35:20] it's just an inactive deployment server right now [19:36:00] keyholder is armed and everything it just isn't the recommended one for deployments for various good reasons I've long since forgotten [19:36:52] I think there was worries about deployers using one host getting confused with changes that were deployed from another host and aren't present on theres [19:36:53] etc. etc. etc. [19:38:17] twentyafterfour, what about an upstream ticket? [19:38:26] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: PHP Fatal Error: Class 'Wikimedia\EquivSet\EquivSet' not found - https://phabricator.wikimedia.org/T189560 (10Reedy) So we shouldn’t be using this job for non master then? [19:39:19] Krenair: I will work on a patch which I will submit upstream [19:39:30] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10Aklapper) I filed (non-public) T198671 about due date handling. [19:39:37] I wish I could WinSCP to deployment-tin so I could download the dumpInterwiki.php generated content. [19:39:48] Hauskatze, why can't you scp to deployment-tin? [19:40:08] WinSCP is different to scp [19:40:14] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Umherirrender) Before a... [19:40:20] but scp should be on windows 10 at least with bash? [19:42:27] I can scp just fine but I haven't tried it on windows [19:43:20] I have not used scp so far [19:44:01] the patch +2 by mmodell about scap interwiki cache seems to work just fine, the script works now (but the usual failure about not having grants on operations/mediawiki-config.git etc) [19:44:20] however git log -n1 --oneline on srv/mediawiki-staging shows a commit of mine :| [19:44:27] and several other untracked stuff [19:45:27] yeah that'll happen if you manage to commit to the repo but not merge the change in the gerrit side [19:45:37] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.22.140) [19:45:45] and actually it'll show that even if you can merge on the gerrit side [19:45:49] will they revert after the next beta-code-update-equiad? [19:45:57] not sure [19:46:00] wouldn't count on it [19:46:01] or beta-scap-equiad [19:46:06] eqiad* :) [19:46:09] I could git-revert [19:46:11] true [19:46:21] mothertongue kicked-in there :) [19:46:33] in english we do it too [19:46:40] but the DC names don't follow english rules [19:47:25] eq is a reference to the host Equinix and iad is a reference to the nearest airport, Washington Dulles International [19:47:58] yup [19:48:07] fwiw [19:48:09] c8e01fdef Revert "Updating interwiki cache" [19:48:09] d58d2ffb7 Updating interwiki cache [19:48:30] git status still shows some stuff but i have not done anything of those so I won't touch it [19:51:13] okay well someone has forgotten about event-schemas again [19:51:17] ignoring that for a moment [19:51:35] jenkins-deploy@deployment-tin:/srv/mediawiki-staging$ git log origin/master..HEAD --oneline [19:51:35] c8e01fdef Revert "Updating interwiki cache" [19:51:35] d58d2ffb7 Updating interwiki cache [19:51:35] jenkins-deploy@deployment-tin:/srv/mediawiki-staging$ [19:51:37] should probably be empty [19:51:51] jenkins-deploy@deployment-tin:/srv/mediawiki-staging$ git reset --hard HEAD^^ [19:51:52] HEAD is now at cb06d95dc Scap: UpdateInterwikiCache fix subclassing [19:53:47] dumb question... how do you 'become' jenkins-deploy? [19:54:13] sudo -u jenkins-deploy -i [19:54:16] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Krenair) [19:54:18] same way you become anyone else [19:54:33] I'm always under my own user. I could 'sudo bash' and becaome root but that's not to be done often I was told [19:54:42] same way the tools 'become' command words [19:54:43] works* [19:54:59] * Krenair shrugs [19:55:01] depends on the system [19:55:08] for some things you need to be root to do anything useful [19:55:08] so I could sudo -u krenair -i ? :P [19:55:18] sure but you shouldn't :P [19:55:28] I have no interest whatsover in do that [19:56:03] roots in prod would be able to do the same to eachother [20:04:17] Krenair: and -i stands for? [20:04:22] 'ignore' ? [20:04:36] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: PHP Fatal Error: Class 'Wikimedia\EquivSet\EquivSet' not found - https://phabricator.wikimedia.org/T189560 (10Daimona) I guess adding Equivset to vendor will be fine after T191740, or if AntiSpoof will be bundled (since it uses equivset as wel... [20:05:01] runs a shell [20:05:17] the shell will be the target user's one [20:05:18] ack [20:19:49] 10Phabricator, 10VisualEditor, 10User-Ryasmeen: 404 on VisualEditor workboard (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10Aklapper) [21:13:35] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: PHP Fatal Error: Class 'Wikimedia\EquivSet\EquivSet' not found - https://phabricator.wikimedia.org/T189560 (10hashar) Ah that is interesting. So mediawiki/vendor on REL branches should only have the dependencies for the extensions in the tarb... [21:14:37] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, and 2 others: Migrate gated extensions jobs to Quibble/Docker - https://phabricator.wikimedia.org/T197469 (10hashar) [21:17:56] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog, 10Epic, 10MobileFrontend (MobileFrontend.js): [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980 (10Jdlrobson) [21:27:44] !log github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator | T199103 [21:27:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:48] T199103: Archive the Transliterator extension - https://phabricator.wikimedia.org/T199103 [21:28:33] Hauskatze: you are my savior on #cleanup :] [21:28:59] hashar: :D [21:29:27] puppet role::maurelio::janitor :P [21:30:27] I watch repository-admins rather than cleanup, but the archiving tasks end there as well too :) [21:34:01] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: PHP Fatal Error: Class 'Wikimedia\EquivSet\EquivSet' not found - https://phabricator.wikimedia.org/T189560 (10Reedy) >>! In T189560#4413972, @hashar wrote: > Ah that is interesting. So mediawiki/vendor on REL branches should only have the dep... [21:44:34] (03CR) 10Hashar: [C: 032] "Argh sorry. At least I confirmed it worked by manually running the job!" [integration/config] - 10https://gerrit.wikimedia.org/r/444884 (https://phabricator.wikimedia.org/T198171) (owner: 10Hashar) [21:46:07] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<22.22%) [21:48:13] !log manually cleaning up disk space on integration-slave-jessie-1002 ( apt-get clean , apt-get autoremove --purge , nuked /srv/jenkins-workspace/workspace/*lint [21:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:54:38] (03CR) 10Hashar: "Is there any specific issue with the jobs?" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/444641 (owner: 10Ejegg) [21:55:04] twentyafterfour i am getting this mkdir: cannot create directory '/srv/phab/images/bb/01': Permission denied [21:55:11] when trying to upload a file [21:55:13] (image) [21:56:08] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [21:56:08] oh [21:56:10] nvm [21:56:43] hashar jenkins has a new shutdown ui :). ie you see this: https://phabricator.wikimedia.org/F23573590 [21:57:09] they also redesigned the login page. [21:59:21] paladox: nice :]]] [21:59:38] we should probably upgrade it [21:59:43] I have lost track of jenkins releases [21:59:52] hashar the new ui for the resgined login page looks really nice! [22:00:03] they should make the whole ui have a update! [22:00:04] "is there a security issue?" -> "yes" -> "upgrade jenkins [22:00:38] pretty much [22:00:55] https://jenkins.io/changelog-stable/ [22:01:24] and the awesome thing is that list issues that got reported for the release [22:01:30] Community reported issues: 1×JENKINS-9104 1×JENKINS-42586 1×JENKINS-37575 1×JENKINS-12 :] [22:08:20] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:11:15] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:12:07] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:12:19] * Krinkle is getting rate-limited at home. [22:12:23] (Phabricator) [22:12:29] First time for me. [22:12:35] Yeah... [22:12:39] Roan was complaining similarly earlire [22:12:44] have we fixed it yet to not apply to get/read-only requests? [22:12:56] Such as avatars and ticket views. [22:13:17] I don't think it's that easy as far as phab is concerned [22:13:29] Cause they seemed to want it for rate limiting crawling bots [22:13:30] I haven't done any writes in the last 4-5 minutes. [22:13:42] Oh wait, right, I'm drafting a comment. I guess those previews also add up. [22:14:48] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review, 10Services (blocked): Update Debian Package for Scap3 to 3.8.4-1 - https://phabricator.wikimedia.org/T199283 (10thcipriani) Adding #Operations for the package update + puppet patch. [22:15:00] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:16:00] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:16:03] 10Scap, 10WorkType-NewFunctionality: Add a --labs option to 'scap update-interwiki-cache' to be able to update the interwiki-labs.php file using Scap - https://phabricator.wikimedia.org/T198844 (10thcipriani) p:05Triage>03Normal [22:16:04] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:16:42] 10Release-Engineering-Team (Next), 10Scap: Add some facility to scap for custom logstash canary checks - https://phabricator.wikimedia.org/T198642 (10thcipriani) p:05Triage>03Normal [22:17:19] 10Release-Engineering-Team (Next), 10Scap: Perform scap canary checks after sync-wikiversions - https://phabricator.wikimedia.org/T198640 (10thcipriani) p:05Triage>03Normal [22:17:56] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:19:11] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:20:25] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:20:50] ^ probably me, checking now [22:23:07] should clear-up once scap updates on deployment-tin pending a few jenkins jobs [22:25:38] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:26:02] (03Abandoned) 10Ejegg: Add --project-branch for fundraising clone prep [integration/config] - 10https://gerrit.wikimedia.org/r/444641 (owner: 10Ejegg) [22:27:44] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:28:34] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:31:17] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:46:13] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:47:07] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:19] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:53:00] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:54:08] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:01] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:25] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:59] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:56:03] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:38] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:06:15] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:07:43] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:08:35] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0]