[00:33:08] 3MediaWiki-Documentation, Continuous-Integration: live examples in mw JSDuck broken on doc.wikimedia.org - https://phabricator.wikimedia.org/T86864#978109 (10Spage) 3NEW a:3hashar [00:48:52] 3Scrum-of-Scrums, operations: Update wikitech wiki with deployment train - https://phabricator.wikimedia.org/T70751#978214 (10bd808) I got poked today as a #scrum-of-scrums pingback to see why this is still open and marked as blocked by #ops. Here's my $0.02: Wikitech is using hetdeploy today as a result of wor... [00:52:41] 3Continuous-Integration, MediaWiki-extensions-Thanks: Thanks needs Jenkins qunit job - https://phabricator.wikimedia.org/T86866#978256 (10Legoktm) [01:16:05] 3MediaWiki-Documentation, Continuous-Integration: live examples in mw JSDuck broken on doc.wikimedia.org - https://phabricator.wikimedia.org/T86864#978314 (10Krinkle) Last week, I ran into the 404 Not Found (on my localhost) and fixed it in https://gerrit.wikimedia.org/r/184558. However it's broken on doc.wikim... [01:16:16] 3MediaWiki-Documentation, Continuous-Integration: live examples in mw JSDuck broken on doc.wikimedia.org - https://phabricator.wikimedia.org/T86864#978317 (10Krinkle) a:5hashar>3Krinkle [01:19:51] (03PS1) 10Krinkle: mediawiki-core-jsduck-publish: Fix resources/ being 404 Not Found [integration/config] - 10https://gerrit.wikimedia.org/r/185109 (https://phabricator.wikimedia.org/T86864) [01:37:13] (03CR) 10Mattflaschen: [C: 04-1] "Duplicate and unexplained dependencies (and if we do need them, the commit message needs to be updated)." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [01:39:02] (03CR) 10Mattflaschen: "Oh, are the dependencies not transitive?" [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [02:42:22] 3Continuous-Integration: Make sure everyone in RelEng has sudo on integration slaves in labs - https://phabricator.wikimedia.org/T86779#978553 (10greg) a:3greg I'll do this when I bored with quarterly review planning. [03:14:42] Project beta-scap-eqiad build #38261: FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38261/ [03:24:40] Project beta-scap-eqiad build #38262: STILL FAILING in 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38262/ [03:35:01] Yippee, build fixed! [03:35:02] Project beta-scap-eqiad build #38263: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38263/ [04:41:27] 3Phabricator, Wikimedia-Git-or-Gerrit: gerritbot subscribing itself to tasks - https://phabricator.wikimedia.org/T86878#978636 (10matmarex) 3NEW [04:48:33] greg-g: I made some changes as to how Jenkins accesses *oid instances but I did verify that nothing changes when puppet ran on those instances [05:10:59] 3Phabricator: "Blocked By" list sort criteria should be documented - https://phabricator.wikimedia.org/T75702#978681 (10scfc) >>! In T75702#777546, @Qgil wrote: > I think they are sorted by most recent activity (tasks with most recent updates would go to top). > > This isn't an issue for most tasks. That case i... [05:20:59] YuviPanda: Can you look at: https://phabricator.wikimedia.org/T86847 before alex arrives? [05:26:12] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #456: FAILURE in 19 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/456/ [05:27:24] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #490: FAILURE in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/490/ [05:28:55] Yippee, build fixed! [05:28:56] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #410: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/410/ [05:51:38] YuviPanda: gotcha, I forget why I pinge dyou on that, but the issue we ran into was a consequence of faidon's ferm rule changes [05:51:57] greg-g: yup. read backscroll, seems to be fine now. [05:52:03] * greg-g nods [05:52:17] kart_: hmm, I’m probably just going to wait till alex shows up for that, I’m afraid [05:52:36] greg-g: I also put a note on your etherpad of things to discuss during all hands :) [05:52:54] :) :) [05:52:55] ie. rebuilding betacluster from scratch and fixing all the differences with prod that way :) [05:53:02] let’s see how that goes. [05:53:20] I’m going to continue trying to remove as many differences as possible in the meantime, though [05:53:37] YuviPanda: it's not a bad idea, and is gaining traction among people [05:53:47] greg-g: yup, and we can call it beta-cluster as well :) [05:54:22] greg-g: built with a rule of *no manual modifications* as well :) and closely mirroring prod setup in every way (we should have enough machines now) [05:54:26] uh, except for lvs and nginx :| [05:54:38] YuviPanda: no worries. [05:56:07] YuviPanda: there will always be differences until we go with full prod mirror :) [05:56:17] anywho, I have a headache and I need sleep [05:56:20] greg-g: indeed, but less differences than we have now [05:56:20] g'night all! [05:56:22] greg-g: good night! [06:36:50] 3Phabricator, Phabricator.org: Phabricator search does not search substrings - https://phabricator.wikimedia.org/T679#11557 (10Qgil) Tested, it works! Phew. [06:37:58] 3operations, Beta-Cluster: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#978747 (10yuvipanda) antimony, our svn server, doesn't actually seem to have LDAP configured, or ssh accessible from outside the cluster. [06:38:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:42:52] 3Phabricator, Phabricator.org: Can't search an exact phrase in Phabricator - https://phabricator.wikimedia.org/T75743#978756 (10Qgil) Unless there is some indexing to be done or some special syntax to be used other than quotes, searching for "whole blocks" gives many results where such string doesn't exist. Gen... [06:43:13] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #407: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/407/ [06:47:50] 3Phabricator: Mingle migration script - https://phabricator.wikimedia.org/T822#978759 (10Qgil) I don't know how much is the fee for keeping the current instance open. Does Mingle offer a way to dump, the contents of a project in some kind of tarball? I guess the team in this situation will maintain the cards f... [06:51:45] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #401: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/401/ [06:58:35] 3Code-Review, Phabricator.org: GitHub->Phabricator bridge for new contributors - https://phabricator.wikimedia.org/T173#978776 (10Qgil) Meanwhile, a small project has been created upstream "Of or pertaining to integration with GitHub." https://secure.phabricator.com/project/profile/582/ [07:03:46] 3Code-Review, Phabricator.org: GitHub->Phabricator bridge for new contributors - https://phabricator.wikimedia.org/T173#978779 (10Qgil) [07:09:47] 3Phabricator: "Blocked By" list sort criteria should be documented - https://phabricator.wikimedia.org/T75702#978791 (10Qgil) If the list is short, I open tasks in new tabs. If the list is long, that smells to tracking task that could use an own project with workboard. Putting the most recently active tasks at... [07:13:29] 3Beta-Cluster: Scap failure on puppetrun on deployment-mediawiki01 - https://phabricator.wikimedia.org/T86883#978792 (10yuvipanda) 3NEW [07:22:15] greg-g: bd808|BUFFER btw, just wanted to remind that Reedy, Chad and Robla can also do LDAP changes, if required :) so not fully blocked on ops [07:26:34] 3Engineering-Community, Phabricator, Wikimedia-Git-or-Gerrit: Migrate Gerrit project ownership request system (+2 rights) to Phabricator - https://phabricator.wikimedia.org/T86639#978816 (10Qgil) [07:27:54] 3Phabricator: Phabricator boards should live update - https://phabricator.wikimedia.org/T129#978819 (10Qgil) [07:33:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:35:49] 3Phabricator, MediaWiki-extensions-OAuth: MediaWiki OAuth dialog text is unclear and sounds more scary than it is - https://phabricator.wikimedia.org/T598#978828 (10Qgil) p:5Low>3Volunteer? We have +1500 users now and most of them have gone through this process silently. The OAuth team claims that what the l... [07:38:28] 3Phabricator: Dashboard for advanced users - https://phabricator.wikimedia.org/T761#978835 (10Qgil) 5Open>3declined a:3Qgil The fact is, advanced users are getting their own dashboard. Declining this task that I created. If someone plans to work on this, feel free to reopen. [07:43:52] 3Phabricator: Lots of unrelated results when searching for specific string - https://phabricator.wikimedia.org/T86805#978842 (10Qgil) Yes, saw that as well. It feels that what happens is that now stemming goes before than exact quote, and therefore you get results that contains strings similar to "/event.gif". I... [07:48:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:10] Project beta-scap-eqiad build #38287: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38287/ [07:53:13] Project beta-scap-eqiad build #38288: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38288/ [07:53:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:54:10] Project beta-scap-eqiad build #38289: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38289/ [07:54:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:01:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:04:19] Project beta-scap-eqiad build #38290: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38290/ [08:04:35] 3Beta-Cluster: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#978863 (10yuvipanda) 3NEW [08:05:52] 3Beta-Cluster: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#978870 (10yuvipanda) [08:05:53] 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#978871 (10yuvipanda) [08:06:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:10:57] 3Beta-Cluster: Scap failure on puppetrun on deployment-mediawiki01 - https://phabricator.wikimedia.org/T86883#978872 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [08:14:20] Project beta-scap-eqiad build #38291: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38291/ [08:24:01] Project beta-scap-eqiad build #38292: STILL FAILING in 0.18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38292/ [08:29:32] (03PS17) 10Adrian Lang: Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [08:33:56] Project beta-scap-eqiad build #38293: STILL FAILING in 0.11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38293/ [08:36:25] 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#978886 (10yuvipanda) [08:36:26] 3Beta-Cluster: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#978885 (10yuvipanda) [08:36:39] 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#973295 (10yuvipanda) [08:36:40] 3Beta-Cluster: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#978863 (10yuvipanda) [08:39:12] (03PS1) 10Hashar: Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 [08:41:09] (03PS2) 10Hashar: Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 [08:41:23] moaar stupid hack [08:44:01] Project beta-scap-eqiad build #38294: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38294/ [08:51:15] (03CR) 10Hashar: [C: 032] Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [08:52:50] (03CR) 10jenkins-bot: [V: 04-1] Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [08:53:47] 3MediaWiki-extensions-ContentTranslation, Continuous-Integration: check that JavaScript mw.hooks are documented using continuous integration - https://phabricator.wikimedia.org/T86544#978914 (10Amire80) p:5Triage>3Low [08:53:55] Project beta-scap-eqiad build #38295: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38295/ [08:54:31] (03PS3) 10Hashar: Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 [08:54:53] (03CR) 10Hashar: "There was a typo in the YAML definition :(" [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [08:55:04] (03CR) 10Hashar: [C: 032] Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [08:55:26] (03CR) 10jenkins-bot: [V: 04-1] Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [09:02:12] (03Merged) 10jenkins-bot: Support injecting an extension in shared job [integration/config] - 10https://gerrit.wikimedia.org/r/185132 (owner: 10Hashar) [09:04:01] Project beta-scap-eqiad build #38296: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38296/ [09:14:02] Project beta-scap-eqiad build #38297: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38297/ [09:18:14] (03PS1) 10Hashar: Add Thanks to the shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/185133 (https://phabricator.wikimedia.org/T86687) [09:18:45] (03CR) 10Hashar: [C: 032] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/185133 (https://phabricator.wikimedia.org/T86687) (owner: 10Hashar) [09:19:39] (03CR) 10jenkins-bot: [V: 04-1] Add Thanks to the shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/185133 (https://phabricator.wikimedia.org/T86687) (owner: 10Hashar) [09:24:04] Project beta-scap-eqiad build #38298: STILL FAILING in 0.18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38298/ [09:25:04] (03Merged) 10jenkins-bot: Add Thanks to the shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/185133 (https://phabricator.wikimedia.org/T86687) (owner: 10Hashar) [09:29:09] (03PS1) 10Hashar: Stop mwext-Thanks-testextension-zend on old branches [integration/config] - 10https://gerrit.wikimedia.org/r/185134 (https://phabricator.wikimedia.org/T86687) [09:33:11] (03CR) 10Hashar: [C: 032] Stop mwext-Thanks-testextension-zend on old branches [integration/config] - 10https://gerrit.wikimedia.org/r/185134 (https://phabricator.wikimedia.org/T86687) (owner: 10Hashar) [09:34:04] Project beta-scap-eqiad build #38299: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38299/ [09:34:47] (03Merged) 10jenkins-bot: Stop mwext-Thanks-testextension-zend on old branches [integration/config] - 10https://gerrit.wikimedia.org/r/185134 (https://phabricator.wikimedia.org/T86687) (owner: 10Hashar) [09:44:02] Project beta-scap-eqiad build #38300: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38300/ [09:53:59] Project beta-scap-eqiad build #38301: STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38301/ [09:55:56] 3Mobile-Web, Continuous-Integration, MediaWiki-extensions-Thanks: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#978962 (10hashar) [09:56:09] 3Mobile-Web, Continuous-Integration, MediaWiki-extensions-Thanks: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#978963 (10hashar) p:5Triage>3High [10:04:07] Project beta-scap-eqiad build #38302: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38302/ [10:14:10] Project beta-scap-eqiad build #38303: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38303/ [10:22:08] (03PS1) 10Hashar: Make sure MW shared job is 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/185144 [10:23:49] (03CR) 10Hashar: [C: 032] "noop" [integration/config] - 10https://gerrit.wikimedia.org/r/185144 (owner: 10Hashar) [10:24:15] Project beta-scap-eqiad build #38304: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38304/ [10:25:13] (03Merged) 10jenkins-bot: Make sure MW shared job is 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/185144 (owner: 10Hashar) [10:34:17] Project beta-scap-eqiad build #38305: STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38305/ [10:37:10] (03CR) 10Hashar: "> I.E. If A depends on B, and B depends on "C, D", does A have to explicitly depend on "B, C, D", or is it enough to depend on B?" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [10:37:15] (03PS5) 10Hashar: Have Thanks depend on MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [10:37:25] (03PS6) 10Hashar: Have Thanks depend on MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [10:39:09] (03CR) 10Hashar: [C: 032] "Updated jobs, thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (owner: 10Mattflaschen) [10:39:32] (03PS7) 10Hashar: Have Thanks depend on MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (https://phabricator.wikimedia.org/T86687) (owner: 10Mattflaschen) [10:39:46] (03CR) 10Hashar: [C: 032] "Added reference to T86687" [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (https://phabricator.wikimedia.org/T86687) (owner: 10Mattflaschen) [10:44:08] Project beta-scap-eqiad build #38306: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38306/ [10:48:20] (03Merged) 10jenkins-bot: Have Thanks depend on MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/183724 (https://phabricator.wikimedia.org/T86687) (owner: 10Mattflaschen) [10:51:43] 3Mobile-Web, Continuous-Integration, MediaWiki-extensions-Thanks: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#979129 (10hashar) I think that is solved, would let you close this task though. Adding a qunit job for thanks is {T86866}. [10:54:02] Project beta-scap-eqiad build #38307: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38307/ [10:55:25] !log https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ is broken since roughly 7:52am UTC. [10:55:29] Logged the message, Master [10:57:00] YuviPanda: the beta scap is broken since ~ 7:50am UTC :-( [11:00:26] AH [11:00:34] it runs /usr/local/bin/sudo-withagent mwdeploy /usr/local/bin/scap [11:00:37] which does ssh-add [11:00:41] and that can't find keys apparently [11:01:53] pfff [11:04:08] Project beta-scap-eqiad build #38308: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38308/ [11:08:01] 3Release-Engineering, Beta-Cluster: beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key - https://phabricator.wikimedia.org/T86901#979139 (10hashar) 3NEW [11:12:25] 3Wikimedia-Labs-Infrastructure, operations, Beta-Cluster: Change mwdeploy homeDirectory field in LDAP from /home/mwdeploy to /var/lib/mwdeploy - https://phabricator.wikimedia.org/T86903#979157 (10hashar) 3NEW [11:14:08] Project beta-scap-eqiad build #38309: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38309/ [11:16:18] 3operations, Beta-Cluster: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#979167 (10hashar) I suspect https://gerrit.wikimedia.org/r/#/c/185127/ changed the mwdeploy user homedir as a side effect, but that might be unrelated. Beta cluster scap is broken: T86901 I ha... [11:24:05] Project beta-scap-eqiad build #38310: STILL FAILING in 0.17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38310/ [11:34:03] Project beta-scap-eqiad build #38311: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38311/ [11:43:57] Project beta-scap-eqiad build #38312: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38312/ [11:53:56] Project beta-scap-eqiad build #38313: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38313/ [12:03:56] Project beta-scap-eqiad build #38314: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38314/ [12:07:47] hashar: heya! I intentionally changed it to /home/mwdeploy [12:08:22] It differs from prod. We should just move the keys to an appropriate place instead [12:08:32] Unfortunately I'm not near my computer now [12:08:39] And won't be for another 6h :( [12:13:54] Project beta-scap-eqiad build #38315: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38315/ [12:21:43] Project beta-update-databases-eqiad build #6918: FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6918/ [12:23:53] Project beta-scap-eqiad build #38316: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38316/ [12:31:39] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:33:54] Project beta-scap-eqiad build #38317: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38317/ [12:35:33] 3Phabricator, Wikimedia-Git-or-Gerrit: gerritbot subscribing itself to tasks - https://phabricator.wikimedia.org/T86878#979305 (10Aklapper) p:5Triage>3Volunteer? Confirming that this is new behavior since T78243 but I don't see a problem with it (yet). [12:43:58] Project beta-scap-eqiad build #38318: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38318/ [12:53:51] Project beta-scap-eqiad build #38319: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38319/ [13:00:25] hashar: around? :) [13:00:37] addshore: hey [13:00:46] hope you are well :) [13:00:53] quick question about your panel on phab https://phabricator.wikimedia.org/W55 [13:01:19] Any chance you can rename it to Authored by Hashar? :D [13:01:38] As I just realised it was on my dashboard but only shows things you have authored ;p [13:01:50] ahh [13:01:55] I need to change the query [13:02:09] Or do that yes! :D [13:02:31] stupid phabricator [13:02:51] we can save a query that is authored: self [13:03:22] I just did a test and I think if you leave the query blank it populates the result with things you have authored [13:03:57] Project beta-scap-eqiad build #38320: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38320/ [13:04:10] https://phabricator.wikimedia.org/W230 [13:04:52] unless for you that just shows things I have authored :D [13:05:23] addshore: fixed! [13:05:31] "my" dashboard is https://phabricator.wikimedia.org/dashboard/manage/6/ [13:05:42] The Authored column should be a query that has no author [13:05:57] Aka the saved query "AuthoredOpen" at https://phabricator.wikimedia.org/maniphest/query/J3Tx4xzOJ51S/ [13:06:41] addshore: are you coming to the MW summit ? [13:06:46] *reloads his dashboard* [13:06:48] yes I am :) [13:07:17] Hah, now that panel has things authored by random people for me :/ [13:09:15] 3Release-Engineering, Wikimedia-Logstash: Upgrade ElasticSearch on logstash boxen to 1.3.6 - https://phabricator.wikimedia.org/T76089#979335 (10hashar) Maybe it can be made to autoupdated? For contint slaves, I wanted to get HHVM automatically upgraded. Faidon recommended to use Debian unattended upgrade mechani... [13:09:39] addshore: guess you will need to build your own [13:10:16] addshore: for the mw summit, I have a couple CI related sessions. Would love to get you there :] https://phabricator.wikimedia.org/T86750 what we did in 2014 || https://phabricator.wikimedia.org/T86752 what we will do in 2015 [13:10:39] the first being a restropective, the second more to discuss [13:12:49] 3Release-Engineering, Wikimedia-Logstash: Upgrade ElasticSearch on logstash boxen to 1.3.6 - https://phabricator.wikimedia.org/T76089#979353 (10Reedy) >>! In T76089#979335, @hashar wrote: > Maybe it can be made to autoupdated? For contint slaves, I wanted to get HHVM automatically upgraded. Faidon recommended to... [13:13:36] Reedy: that is about upgrading ElasticSearch on production right ? [13:13:59] Project beta-scap-eqiad build #38321: STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38321/ [13:14:45] yeah [13:14:50] I guess the bug isn't fully clear :) [13:15:06] 3Release-Engineering, Wikimedia-Logstash: Upgrade ElasticSearch on production logstash boxen to 1.3.6 - https://phabricator.wikimedia.org/T76089#979354 (10Reedy) [13:15:07] fixed :P [13:15:48] 3Release-Engineering, Wikimedia-Logstash: Upgrade ElasticSearch on production logstash boxen to 1.3.6 - https://phabricator.wikimedia.org/T76089#979355 (10hashar) I thought this task was about logstash on the beta cluster where unattended upgrade might have made sense. Disregard my earlier comment :] [13:16:36] hashar: you should see me there :) [13:22:49] (03PS1) 10Hashar: Add mediawiki-extensions-hhvm to 3 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/185165 [13:23:43] (03CR) 10Hashar: [C: 032] Add mediawiki-extensions-hhvm to 3 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/185165 (owner: 10Hashar) [13:23:52] Project beta-scap-eqiad build #38322: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38322/ [13:24:17] Reedy: and the beta scap job is broken because ssh-add can no more find the mwdeploy ssh key :( [13:24:20] the home got changed [13:24:39] Did Yuvi change that? [13:25:00] (03Merged) 10jenkins-bot: Add mediawiki-extensions-hhvm to 3 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/185165 (owner: 10Hashar) [13:25:09] I filled a bug for it earlier https://phabricator.wikimedia.org/T86903 [13:33:40] * hashar whistles [13:34:01] Project beta-scap-eqiad build #38323: STILL FAILING in 0.35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38323/ [13:40:03] (03PS1) 10Hashar: Add five extensions to shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/185167 [13:42:49] 3Wikimedia-Labs-Infrastructure, operations, Beta-Cluster: Change mwdeploy homeDirectory field in LDAP from /home/mwdeploy to /var/lib/mwdeploy - https://phabricator.wikimedia.org/T86903#979419 (10Reedy) I blame @yuvipanda ``` 07:19 YuviPanda: set home of mwdeploy to /home/mwdeploy in LDAP ``` [13:43:18] (03CR) 10Hashar: [C: 032] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/185167 (owner: 10Hashar) [13:43:25] bad YuviPanda [13:43:54] Project beta-scap-eqiad build #38324: STILL FAILING in 0.18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38324/ [13:50:03] Reedy: why is it different from prod? [13:50:09] Reedy: also you can unfix it! You have the perms [13:50:12] And I'm still out [13:50:25] Might have to restart nscd [13:50:56] fuck ldap [13:51:05] YuviPanda: Not sure. hashar will likely know [13:51:14] (03Merged) 10jenkins-bot: Add five extensions to shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/185167 (owner: 10Hashar) [13:51:45] YuviPanda: I am not sure the sillyshell -> bash change for mwdeploy cause the issue [13:51:50] but seems quite likely [13:52:18] hashar: no I manually modified homedir [13:52:26] OH MY GOD [13:52:30] :D [13:52:43] The cherry pick that was on salt had a note saying [13:52:55] HACK: need ops to modify LDAP entry [13:52:58] YuviPanda: I think mwdeploy ssh key is in /var/lib/mwdeploy because it should not be on the shared homem [13:53:03] So I did and modified the local app [13:53:17] hashar: can't you specify the location to ssh-add [13:53:29] Or let the home be same in prod [13:53:55] Project beta-scap-eqiad build #38325: STILL FAILING in 0.18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38325/ [13:54:43] YuviPanda: what was the local hack ? [13:54:58] maybe we can restore it? [13:55:14] It set the mwdeploy's homedir [13:55:18] In puppet [13:55:31] ah [13:55:44] Need to do that AND have the LDAP change [13:55:48] Reedy has perms [13:55:55] but since the user is in LDAP [13:55:59] it is not created locally on the instance [13:56:04] so the puppet change is not reflected [13:56:21] Yup you need to modify in LDAP and set it in puppet [13:56:48] so that is blocked on https://phabricator.wikimedia.org/T86903 to change the LDAP entry :° [13:56:58] not sure whether Reedy can change the virt1000 ldap though [13:57:18] He can [13:57:24] I checked earlier [13:57:59] Same as terbium [13:58:01] I can change some ldap stuff [13:58:02] Yeah [13:58:08] ldap is scary though :P [13:58:12] Hehe [13:58:20] can you pick https://phabricator.wikimedia.org/T86903 so ? :] [13:58:21] Or you can wait for... 3h [13:58:23] And I'll be back [13:58:28] there is a puppet hack to change mwdeploy homedir [13:58:40] but it is not applied by puppet cause the mwdeploy user already exist (via ldap lookup) [13:58:56] I really want to unify them tho. Will find a solution for ssh-add [13:58:58] YuviPanda: thanks for showing up. I am sure Reedy can figure out the proper LDAP commands hehe [13:59:09] YuviPanda: and huge thanks for all the beta related cleanup ! [13:59:16] ldap is awful [13:59:22] +1 [13:59:28] that is why we have sysadmins [13:59:38] Reedy: I installed ldapvi on terbium. Makes it less awful [13:59:41] To edit [13:59:43] Slightly [13:59:43] so we can fill a Task and have knowledgeable people figure it out for us [13:59:51] I sadly have to disappear again [13:59:53] I'll be brb [13:59:55] YuviPanda: Useless [13:59:56] In 3h [14:00:07] Sorry. [14:03:55] Project beta-scap-eqiad build #38326: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38326/ [14:05:28] Reedy: maybe opsen can help? [14:05:37] though I have seen Coren / AndrewBoggot recently [14:13:57] Project beta-scap-eqiad build #38327: STILL FAILING in 0.17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38327/ [14:22:26] 3VisualEditor, Continuous-Integration, MediaWiki-extensions-Flow: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#979486 (10hashar) 3NEW [14:22:49] 3MW-RFCs, ArchCom, Continuous-Integration: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#979493 (10hashar) [14:22:50] 3VisualEditor, Continuous-Integration, MediaWiki-extensions-Flow: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#979494 (10hashar) [14:23:52] Project beta-scap-eqiad build #38328: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38328/ [14:33:53] Project beta-scap-eqiad build #38329: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38329/ [14:43:55] Project beta-scap-eqiad build #38330: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38330/ [14:49:41] I am off, probably for the rest of the day [14:49:42] got meeting [14:50:17] !log beta-scap-eqiad broken since ~ 7:52am UTC. Depends on mwdeploy user homedir to be fixed in LDAP https://phabricator.wikimedia.org/T86903 [14:50:22] Logged the message, Master [14:51:06] 3Scrum-of-Scrums, operations: Update wikitech wiki with deployment train - https://phabricator.wikimedia.org/T70751#979542 (10Andrew) I have no objection to allowing deployment access, at this point. What I'd /really/ like to do, though, is have wikitech hosted on the normal cluster rather than on virt1000. I'... [14:53:52] Project beta-scap-eqiad build #38331: STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38331/ [14:54:20] 3Scrum-of-Scrums, operations: Update wikitech wiki with deployment train - https://phabricator.wikimedia.org/T70751#979547 (10Joe) We want it to be hosted on a separate host from the puppetmaster, we don't want it to be on the mediawiki cluster at all. Wikitech offers, if any, a completely different surface of a... [14:55:31] 3Scrum-of-Scrums, operations: Update wikitech wiki with deployment train - https://phabricator.wikimedia.org/T70751#979548 (10Andrew) That's fine with me as well. [15:04:01] Project beta-scap-eqiad build #38332: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38332/ [15:13:56] Project beta-scap-eqiad build #38333: STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38333/ [15:23:56] Project beta-scap-eqiad build #38334: STILL FAILING in 0.17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38334/ [15:33:54] Project beta-scap-eqiad build #38335: STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38335/ [15:36:38] remember, labs-wide outage starts in 2.5 hours [15:43:52] Project beta-scap-eqiad build #38336: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38336/ [15:53:52] Project beta-scap-eqiad build #38337: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38337/ [15:59:43] 3MediaWiki-ResourceLoader, operations, MediaWiki-Core-Team: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#979621 (10matmarex) [16:00:57] 3MediaWiki-ResourceLoader, operations, MediaWiki-Core-Team: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#979631 (10Jdforrester-WMF) It's an over-optimisation in ResourceLoader which creates a bug exposed by the way we deploy. [16:04:32] Project beta-scap-eqiad build #38338: STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38338/ [16:13:59] Project beta-scap-eqiad build #38339: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38339/ [16:23:53] Project beta-scap-eqiad build #38340: STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38340/ [16:24:32] ^d: Gosh. When did we unlock visibility of Diffusion? [16:24:46] <^d> Code deploy yesterday unbroke it [16:25:42] ^d: Aha. Nice! [16:25:58] <^d> We should be able to finish the remaining repos out now [16:26:42] 3Phabricator: Fix search in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#979672 (10Chad) [16:26:45] 3Phabricator, Phabricator.org: Can't search an exact phrase in Phabricator - https://phabricator.wikimedia.org/T75743#979670 (10Chad) 5Resolved>3Open Ugh, this worked in my testing. Will have to revisit. [16:27:01] 3Continuous-Integration: Add Babel, cldr and UniversalLanguageSelector extensions to mediawiki-phpunit-(zend|hhvm) - https://phabricator.wikimedia.org/T86930#979673 (10Nemo_bis) 3NEW [16:29:04] ^d: VE. VE! VE!! [16:29:08] * James_F coughs. [16:29:18] <^d> Might want to get that couch checked out. [16:29:21] <^d> Seems nagging ;-) [16:29:34] * James_F laughs. [16:33:58] Project beta-scap-eqiad build #38341: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38341/ [16:36:30] chrismcmahonbrb: any jobs we should just turn off during the labs outage? [16:37:52] wtf, https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [16:43:56] Project beta-scap-eqiad build #38342: STILL FAILING in 0.17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38342/ [16:46:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:23] Not cool. [16:47:13] Dear Deployment folks, do I need SWAT for small change? https://gerrit.wikimedia.org/r/#/c/185196/ [16:47:57] 3Release-Engineering, Beta-Cluster: beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key - https://phabricator.wikimedia.org/T86901#979736 (10greg) p:5Triage>3Unbreak! Adding #Deployment-Systems @mmodell: can you take a look at this, please? @yuvipanda can probably help figure out what change b... [16:48:23] <^d> kart_: Yeah I'd put that on evening swat unless it's causing problems actively. [16:48:38] what he said ^ [16:48:41] ^d: blocker :) [16:48:44] but fine [16:49:10] ^d: It looks I missed today's window. [16:49:16] * kart_ checks [16:49:43] ahh, that one's easy, is it causing "untrusted" errors? [16:49:44] <^d> kart_: Missed the AM, but the evening one is in ~7h [16:50:07] here's how alex sold it to: [16:50:08] 11:49 < akosiaris> greg-g: https://gerrit.wikimedia.org/r/185196 is a very simple change in the configuration for the ContentTranslation extension. It does not work on many browsers without it. When is the best time to deploy it ? [16:50:12] :P [16:50:19] the "it does not work on many browsers without it" part :) [16:50:39] greg-g: yep [16:51:05] Project beta-scap-eqiad build #38343: STILL FAILING in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38343/ [16:51:14] greg-g: I don't think it's worth turning jobs off and back on again [16:52:04] chrismcmahon: how long does it take to turn them off/on? [16:52:43] greg-g: I just have to go into each one in Jenkins, maybe what 30 or so? They can run, it won't hurt anything [16:53:24] greg-g: besides, I want to show them running to Elena later on today [16:53:33] we should probably silence wmf-insecte at least for the FAILURE spam [16:53:53] Project beta-scap-eqiad build #38344: STILL FAILING in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38344/ [16:54:23] greg-g: could be. I don't think the browser test failures will contribute much spam, but the other jobs might [16:54:43] (because the browser test jobs run very slowly only 2 at a time) [16:55:00] 3Release-Engineering, Beta-Cluster: beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key - https://phabricator.wikimedia.org/T86901#979751 (10greg) btw, I'm turning off the beta code update job because: A) it's been spamming forever STILL FAILING B) the wmf labs filesystem outage starts in 1 hour an... [16:56:09] * greg-g just disabled beta-scap-update [16:56:19] s/update/eqiad/ but you know what I mean :) [16:57:33] $HOME for mwdeploy changed to /var/lib/mwdeploy quite a while ago I thought. [16:58:00] oh the bug is that it changed back? [16:58:33] I wonder if that has something to do with the ldap changes that Yuvi made? [17:00:24] the quick hack for that is just to copy the ssh key on deployment-bastion [17:01:42] bd808: comment on bug plz :) (beta will be down for a while anyways today due to the labs outage, I'm not *that* worried on timing) [17:02:15] (yes, I set to unbreak now!, I can reset to high probably given the labs outage) [17:02:24] 3Release-Engineering, Beta-Cluster: beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key - https://phabricator.wikimedia.org/T86901#979768 (10bd808) ``` [09:57] < bd808> $HOME for mwdeploy changed to /var/lib/mwdeploy quite a while ago I thought. [09:58] < bd808> oh the bug is that it change... [17:05:29] 3Release-Engineering, Beta-Cluster: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#979769 (10greg) 3NEW [17:06:52] !log turned off the beta-scap-eqiad jenkins job due to the persistent failing (https://phabricator.wikimedia.org/T86901) and the impending labs outage [17:06:54] Logged the message, Master [17:08:59] 3Release-Engineering, Beta-Cluster: beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key - https://phabricator.wikimedia.org/T86901#979785 (10greg) See also: {T86903} [17:22:42] 3Phabricator, MediaWiki-extensions-OAuth: MediaWiki OAuth dialog text is unclear and sounds more scary than it is - https://phabricator.wikimedia.org/T598#979813 (10csteipp) [17:24:58] 3Continuous-Integration, Mobile-Web, MediaWiki-extensions-Thanks: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#979821 (10Jdlrobson) Thanks @hashar Yup so once https://gerrit.wikimedia.org/r/#/c/185093/ is fixed than Thanks will have its first QUnit test. T... [17:43:57] 3Deployment-Systems, MediaWiki-Core-Team, Librarization: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#979878 (10csteipp) After https://github.com/FriendsOfPHP/security-advisories/pull/44 was merged, I reran the jenkins build and was notified about the updat... [17:47:38] 3Deployment-Systems, MediaWiki-Core-Team, Librarization: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#979889 (10greg) >>! In T74193#979878, @csteipp wrote: > Should I have Jenkins email security@ instead of just me? +1 to redundancy [17:55:01] 3Phabricator: Mingle migration script - https://phabricator.wikimedia.org/T822#979904 (10Awjrichards) >>! In T822#978759, @Qgil wrote: > I don't know how much is the fee for keeping the current instance open. We should double check with @Eloquence but iirc, Thoughtworks is offering hosted Mingle for free for u... [17:58:56] 3Phabricator: phabricator - don't run as root - https://phabricator.wikimedia.org/T77976#979918 (10Krenair) [18:15:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:53] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:53] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:54] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:15:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:59] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #352: FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/352/ [18:16:03] PROBLEM - App Server bits response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:05] PROBLEM - App Server bits response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:05] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #408: FAILURE in 3 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/408/ [18:16:05] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #480: FAILURE in 2 min 42 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/480/ [18:16:05] annnd the spam begins [18:16:05] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:16:05] YuviPanda: easy way to silence shinken-wm during the outage? [18:16:05] RECOVERY - App Server bits response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.002 second response time [18:16:06] greg-g: /mode #wikimedia-releng +q shinken-wm ? [18:16:06] PROBLEM - HHVM Queue Size on deployment-mediawiki02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [80.0] [18:16:06] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:16:06] * greg-g nods [18:16:06] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:16:06] PROBLEM - HHVM Queue Size on deployment-mediawiki01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [80.0] [18:17:06] now, remind me to -q when it's over :) [18:20:33] greg-g: chrismcmalunch bd808 mea culpa [18:20:35] also I’m back [18:23:55] YuviPanda: no worries [18:26:38] o/ YuviPanda [18:26:57] any idea where to start on https://phabricator.wikimedia.org/T85013 (monitor redis list length)? [18:27:39] is that for prod or for labs/ [18:27:56] also can someone explain why exactly mwdeploy needs to have home set to /var/lib/mwdeploy? [18:28:00] it’s /home/mwdeploy in prod [18:28:56] YuviPanda: Oh? Prod switched it? We just need to install the custom ssh key in the right place. Doesn't so much matter where [18:29:17] bd808: yeah, I looked at the LOCAL HACK, and it had it set to /var/lib/mwdeploy, and prod’s was /home/mwdeploy [18:29:18] I thought I remebered that it switched the other direction [18:29:31] ah. gotcha [18:29:33] bd808: so I looked at /home/mwdeploy and at /var/lib/mwdeploy, found them empty, and then just switched them in LDAP [18:29:47] looks like I didn’t use ls -a, or looked in the wrong places [18:29:48] you forgot to look for .ssh [18:30:00] also I ignored wmf-insecte about… a year ago? or whenever it got introduced [18:30:12] so I assumed something will bark here if I fuck up, and completely forgot about the ignore [18:30:14] it's their in both but .ssh in the old location also has an id_rsa key [18:30:15] YuviPanda: browser tests love you, too [18:31:31] greg-g: heh :) I’ve unignored it now [18:31:54] bd808: anyway, I think fix is to ignore /var/lib/mwdeploy and put key in /home/mwdeploy [18:32:06] which we can’t do atm because of NFS maintenance [18:32:08] *nod* [18:32:30] I could technically sneak into the NFS server and do it, but eh.. [18:32:52] Well just fix the puppet config and things will work out eventually [18:33:03] bd808: I fixed the local hack too, yeah [18:33:15] well [18:33:16] not really [18:33:18] * YuviPanda fixes [18:33:19] I think the key is provisioned in beta::scap::master [18:34:17] yup [19:15:41] what happened to wikibugs? [19:16:15] oh blah [19:16:18] I probably have to touch it [19:16:54] tools.wikibugs@tools-trusty:~/wikibugs2$ touch channels.yaml [19:16:54] touch: cannot touch ‘channels.yaml’: Permission denied [19:17:06] greg-g: I guess it's half down until labs is no longer read-only [19:27:10] legoktm: I can add another hack :P [19:28:02] valhallasw`cloud: how will you add a hack… when you cannot write to the filesystem? [19:28:05] * YuviPanda zips valhallasw`cloud’s mouth [19:28:26] YuviPanda: by fiddling bits... IN MEMORY! [19:28:48] valhallasw`cloud: if only it was lisp :P) [19:30:03] oh my [19:30:12] do ittttt [19:35:21] legoktm: I was thinking; maybe we should just let it join on first message to a channel? [19:35:54] legoktm: except for -labs because -labs is awesome [19:36:02] :D [19:36:11] legoktm: or you can put a sleep(1) before channel joins :) [19:36:27] YuviPanda: no, not really. It has to be a sleep(n) *between* joins :P [19:37:07] valhallasw`cloud: before joins is a subse of between joins, no? [19:37:21] valhallasw`cloud: yeah, that should fix it [19:39:04] valhallasw`cloud: https://gerrit.wikimedia.org/r/185227 [19:39:22] oh that's better than my version [19:41:12] legoktm: https://gerrit.wikimedia.org/r/#/c/185227/ ;-) [19:41:38] legoktm: but let me test something first [19:42:14] valhallasw`cloud: we probably could get rid of the autojoin entirely [19:42:23] oh you did that :D [19:42:27] legoktm: it's useful to see it reconnected to -labs [19:42:32] agreed [19:44:32] ok, join-and-immediate-msg is safe [19:44:44] let's see what git pull says :P [19:46:31] but of course, PHP is like 'Hey, I can't exec() but I'm not going to tell you LOLOLOLOL' [19:47:03] I'm curious as to why you used PHP and not python [19:47:25] because it required zero setup [19:47:46] UWSGI! [19:47:51] REQUIRES ZERO SETUP TOO! :P [19:48:01] YuviPanda: last time I tried, which was last week, it was significantly more than zero setup [19:48:14] #/usr/bin/python3\nimport subprocess; print(subprocess.check_output(['git', 'pull'])) [19:48:31] even thinking about 'god, which directories do I need to make??' is more effort than valhallasw`cloud: we fixed lots of issues since :P [19:48:58] legoktm: that might work, but I'm not sure if it does. Does lighttpd execute .py files if they are chmod 755? [19:49:04] do they need to be called .cgi instead? [19:49:09] do they need to be in cgi-bin? [19:49:11] the questions! [19:50:41] need to be in cgi-bin and just a .py extension and executable [19:50:52] you can look at legobot's cgi-bin for examples :P [19:50:59] mkdir www/python/src [19:51:00] is this cgi-bin /in/ public-html or in ~? [19:51:00] done [19:51:07] in public_html [19:51:09] I think. [19:51:21] see how php is easier for something like this ;-) [19:51:22] yes [19:51:37] YuviPanda: and then some magic webserver2 incantation [19:57:50] IT'S A TRAP! [19:57:50] yup, yup [19:57:50] one step at a time [19:57:50] magic flask in a box [19:57:50] YuviPanda: (I'm all for WSGI, but sometimes PHP /is/ the easier solution. Unfortunately :( ) [19:57:50] true, but we can fix that [19:57:50] \o/ [19:57:51] let’s just make PHP Shit! [19:57:51] let's remove the default php fcgi [19:57:51] :P [19:57:51] * valhallasw`cloud loves to see the world burn [19:57:51] There's a token for that! [19:57:52] * YuviPanda sees no point in quiddity [19:57:52] * YuviPanda files bug to disable quiddity [19:57:52] D: [20:02:00] bd808: greg-g labs is back! [20:02:40] YuviPanda: already? that was fast [20:02:55] chrismcmahon: it was an abort, sadly [20:02:57] coren is writing email [20:02:58] now [20:03:05] ah, got it [20:09:56] YuviPanda: yay [20:10:13] hacking around to fix scap now [20:10:42] RECOVERY - HHVM Queue Size on deployment-mediawiki02 is OK: OK: Less than 30.00% above the threshold [10.0] [20:10:50] wb shinken-wm [20:13:32] greg-g: chrismcmahon can either of you force a scap? [20:13:36] * YuviPanda isn’t aware of how to [20:14:57] oh, I turned it off, one second [20:15:08] the jenkins job [20:15:56] runing [20:16:07] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/38345/ [20:16:19] Identity added: /home/mwdeploy/.ssh/id_rsa (/home/mwdeploy/.ssh/id_rsa) [20:29:04] (03PS1) 10Legoktm: Whitelist XZise (pywikibot maintainer with +2) [integration/config] - 10https://gerrit.wikimedia.org/r/185219 [20:29:09] alright, the team member slide is done, geez [20:30:26] (03PS2) 10Krinkle: mediawiki-core-jsduck-publish: Fix resources/ being 404 Not Found [integration/config] - 10https://gerrit.wikimedia.org/r/185109 (https://phabricator.wikimedia.org/T86864) [20:38:02] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #407: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/407/ [20:38:18] greg-g: chrismcmahon hmm, that scap’s been going on for… 22 mins [20:38:20] is that normal? [20:38:32] 20:36:56 20:36:56 Finished mw-update-l10n (duration: 17m 29s) [20:38:32] yeah [20:38:40] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/38345/console [20:38:44] oh ok [20:39:01] stupid languages [20:45:13] 3Phabricator: Fail gracefully if cookies disabled - currently a 503 is shown - https://phabricator.wikimedia.org/T267#980240 (10Aklapper) [20:45:49] Project beta-scap-eqiad build #38345: STILL FAILING in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38345/ [20:46:51] what! [20:46:57] Build timed out (after 30 minutes). Marking the build as failed. [20:47:01] muther fasdflaskdfjlasdk;;fjsa;ld [20:47:55] scap-rebuild-cdbs: 57% (ok: 4; fail: 0; left: 3).... so close! [20:47:59] :( [20:49:01] Yippee, build fixed! [20:49:01] Project beta-scap-eqiad build #38346: FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38346/ [20:49:23] finally [20:55:08] greg-g: it seems to have failed [20:55:16] greg-g: but I suppose this isn’t the same failure you were seeing earlier? [20:57:25] YuviPanda: you sure? that failed due to timeout, but then it went again and finished, what are you seeing? [20:57:41] I’m looking at https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/38345/console [20:57:49] and that failed due to a timeout [20:57:59] but the earlier failures were just nothing working at all because of ssh issues, no? [20:58:17] yep https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/38344/console [20:58:26] greg-g: oh [20:58:33] greg-g: I see what you mean [20:58:36] so it actually works now [20:59:02] greg-g: shall I close https://phabricator.wikimedia.org/T86901 now [21:00:27] suppose so! [21:00:42] that's why I pasted: 15:16 <+ greg-g> Identity added: /home/mwdeploy/.ssh/id_rsa (/home/mwdeploy/.ssh/id_rsa) [21:00:45] :) [21:01:02] (is that the right location? or did we just live hack that to work?) [21:02:33] greg-g: it’s the same location as prod [21:02:38] as to wether it’s the ‘right’ location... [21:02:42] that’s still under discussion [21:03:47] (03CR) 10Krinkle: [C: 032] mediawiki-core-jsduck-publish: Fix resources/ being 404 Not Found [integration/config] - 10https://gerrit.wikimedia.org/r/185109 (https://phabricator.wikimedia.org/T86864) (owner: 10Krinkle) [21:04:16] oh [21:10:32] 3operations, Beta-Cluster, Wikimedia-Labs-Infrastructure: Change mwdeploy homeDirectory field in LDAP from /home/mwdeploy to /var/lib/mwdeploy - https://phabricator.wikimedia.org/T86903#980386 (10hashar) >>! In T86903#980010, @yuvipanda wrote: > It's /home/mwdeploy in prod, should be /home/mwdeploy in beta. Yea... [21:11:31] (03Merged) 10jenkins-bot: mediawiki-core-jsduck-publish: Fix resources/ being 404 Not Found [integration/config] - 10https://gerrit.wikimedia.org/r/185109 (https://phabricator.wikimedia.org/T86864) (owner: 10Krinkle) [21:13:39] 3Deployment-Systems: [scap] Log directly to logstash via redis input - https://phabricator.wikimedia.org/T86969#980403 (10bd808) 3NEW [21:14:54] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#980413 (10hashar) Can we get the job defined with Jenkins Job Builder in `integration/config.git`. Multiple people should be able to help doing the convers... [21:16:31] 3MediaWiki-extensions-Thanks, Continuous-Integration: Thanks needs Jenkins qunit job - https://phabricator.wikimedia.org/T86866#980426 (10hashar) I might be able to bring up a qunit job that fetches Thanks dependencies. Migrating the qunit job to a be a shared job among multiple repositories is a different hors... [21:23:45] 3Beta-Cluster: Cannot open VE in Betalabs , throwing error Error loading data from server: 503: parsoidserver-http: HTTP 503 - https://phabricator.wikimedia.org/T86951#980467 (10Krenair) [21:23:59] 3Beta-Cluster: Cannot open VE in Betalabs , throwing error Error loading data from server: 503: parsoidserver-http: HTTP 503 - https://phabricator.wikimedia.org/T86951#980474 (10Krenair) a:3Arlolra [21:24:57] 3Beta-Cluster: Cannot open VE in Betalabs , throwing error Error loading data from server: 503: parsoidserver-http: HTTP 503 - https://phabricator.wikimedia.org/T86951#980477 (10Krenair) 5Open>3Resolved Parsoid service had stopped for some reason, @Arlolra fixed it [21:25:03] 3MediaWiki-extensions-Flow, Beta-Cluster, Release-Engineering: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#980480 (10hashar) The [[ https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ | beta-update-databases-eqiad ]] Jenkins job is a 'multiple config... [21:29:27] 3Deployment-Systems: [scap] Log directly to logstash via redis input - https://phabricator.wikimedia.org/T86969#980503 (10greg) p:5Triage>3High [21:29:29] 3Deployment-Systems: [scap] Log directly to logstash via redis input - https://phabricator.wikimedia.org/T86969#980403 (10greg) p:5High>3Normal [21:31:11] 3Deployment-Systems, MediaWiki-Core-Team, operations, Release-Engineering: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#980513 (10greg) >>! In T1342#974944, @Reedy wrote: > https://gerrit.wikimedia.org/r/#/c/184817/ was merged yesterday, we good here? [21:32:39] 3MediaWiki-extensions-CLDR, MediaWiki-extensions-UniversalLanguageSelector, MediaWiki-extensions-Babel, Continuous-Integration: Add Babel, cldr and UniversalLanguageSelector extensions to mediawiki-phpunit-(zend|hhvm) - https://phabricator.wikimedia.org/T86930#980523 (10hashar) Thanks @Nemo_bis This can be done... [21:38:28] 3MediaWiki-extensions-CLDR, MediaWiki-extensions-UniversalLanguageSelector, MediaWiki-extensions-Babel, Continuous-Integration: Add Babel, cldr and UniversalLanguageSelector extensions to mediawiki-phpunit-(zend|hhvm) - https://phabricator.wikimedia.org/T86930#980540 (10hashar) I would love to have all the exten... [21:38:52] 3MediaWiki-extensions-CLDR, MediaWiki-extensions-UniversalLanguageSelector, MediaWiki-extensions-Babel, Continuous-Integration: Add Babel, cldr and UniversalLanguageSelector extensions to mediawiki-phpunit-(zend|hhvm) - https://phabricator.wikimedia.org/T86930#980543 (10hashar) [21:38:53] 3MW-RFCs, Continuous-Integration, ArchCom: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#980542 (10hashar) [21:39:15] 3MW-RFCs, Continuous-Integration, ArchCom: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#23797 (10hashar) More extensions are being added. We still have to migrate the qunit jobs to be shared as well. [21:40:35] 3MediaWiki-extensions-Flow, VisualEditor, Continuous-Integration: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#980549 (10hashar) 5Open>3Resolved a:3hashar Fixed by Gerrit #185175. The Flow extension is now included in the shared PHPUnit job. Thanks! [21:40:36] 3MW-RFCs, Continuous-Integration, ArchCom: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#980552 (10hashar) [21:43:27] greg-g: https://phabricator.wikimedia.org/T86901 should be done, no? not externally blocked? [21:43:30] greg-g: considering that it works [21:45:01] YuviPanda: well given the debate never came to a solution in -operations... [21:45:32] greg-g: that’s a separate issue, about which is ‘correct’. Doesn’t have any practical differences. I’ve filed https://phabricator.wikimedia.org/T86971 to track that [21:45:49] greg-g: I’m writing an explanatory comment on T86901 now [21:46:28] k [21:46:35] do as you deem reasonable [21:46:37] (03PS1) 10Krinkle: mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) [21:46:43] I really should stop procrastinating [21:46:49] greg-g: will do [21:46:52] * YuviPanda sends greg-g hugs [21:46:54] :) [21:47:28] (03PS2) 10Hashar: Whitelist XZise (pywikibot maintainer with +2) [integration/config] - 10https://gerrit.wikimedia.org/r/185219 (owner: 10Legoktm) [21:47:30] maybe we need hard numbers.. data driven approach [21:47:38] how many changes get cherry-picked on beta master [21:47:44] how long would they have waited for ops [21:47:48] if there was only one master [21:49:08] (03CR) 10Hashar: [C: 032] Whitelist XZise (pywikibot maintainer with +2) [integration/config] - 10https://gerrit.wikimedia.org/r/185219 (owner: 10Legoktm) [21:49:56] (03Merged) 10jenkins-bot: Whitelist XZise (pywikibot maintainer with +2) [integration/config] - 10https://gerrit.wikimedia.org/r/185219 (owner: 10Legoktm) [21:50:24] mutante: yeah, good idea... /me snaps his fingers for the metrics genie to appear and do that for him [21:50:38] analytics .. ? [21:50:49] they always say no :( [21:50:58] * greg-g pouts in a comical way [21:51:03] ok, going for real [21:51:04] i understand [21:51:20] there is also Quim's tool [21:51:43] well, technically i learned it's outsourced [21:51:53] korma.wmflabs.org [21:51:56] well, a count of ‘cherry picks on beta cluster’ can’t be done by analytics or korma [21:52:07] why not [21:52:11] since the contents of betacluster cherry-picks isn’t public [21:52:15] and exists only on deployment-salt [21:52:24] they also do other non-public things [21:52:27] so what you really need is a diamond collector that runs on deployment-prep [21:52:33] and reports to graphite [21:52:38] YuviPanda: I was joking, really [21:52:42] i might be naive, but analyzing data.. and analytics [21:52:49] seems natural [21:52:50] mutante: yes, that *is* naive :) [21:53:02] analytics doesn’t do anything related to our graphite data, for example. [21:53:17] greg-g: you should go for real :) [21:53:21] i'll skip my next comment [21:53:26] so you're telling me I shouldn't come to you all for my next surgical operation? [21:53:46] * YuviPanda sticks a bamboo in greg-g’s neck [21:55:13] this clearly needs another meeting,.. cough [21:56:39] (don't tempt me) [21:56:48] (/me is now just waiting for the ECT quarterly review) [21:57:25] (03CR) 10Hashar: "I have deployed the change. Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/185219 (owner: 10Legoktm) [21:59:35] 3MediaWiki-extensions-Flow, Collaboration-Team, Beta-Cluster, Release-Engineering: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#980614 (10EBernhardson) [22:00:02] 3MediaWiki-extensions-Flow, Collaboration-Team, Beta-Cluster, Release-Engineering: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#980617 (10EBernhardson) a:3EBernhardson [22:00:13] 3Phabricator: Migration of Analytics Research & Data to Phabricator - https://phabricator.wikimedia.org/T826#980619 (10ggellerman) [22:00:14] 3Phabricator, Phabricator.org: Assign due date to task and have tasks with due dates displayed on a calendar (similar to Trello) - https://phabricator.wikimedia.org/T76094#980620 (10ggellerman) [22:01:44] 3Phabricator, Phabricator.org: Assign due date to task and have tasks with due dates displayed on a calendar (similar to Trello) - https://phabricator.wikimedia.org/T76094#789403 (10ggellerman) R&D also receives requests with a fixed delivery date and we would need to have this functionality in Phab to facilitat... [22:03:42] 3MediaWiki-extensions-Flow, Collaboration-Team, Beta-Cluster, Release-Engineering: beta-update-databases-eqiad failing on enwiki - https://phabricator.wikimedia.org/T86934#980633 (10EBernhardson) this looks to have been caused by gerrit 165683, the maint script that goes with it(which will be run in prod eventua... [22:04:49] 3MediaWiki-extensions-Flow, VisualEditor, Continuous-Integration: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#980635 (10Mattflaschen) 5Resolved>3Open That's not merged. We're still discussing how to do it. [22:04:50] 3MW-RFCs, Continuous-Integration, ArchCom: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#980637 (10Mattflaschen) [22:05:08] 3Phabricator: Compress phabricator assets (favicon, CSS, JS etc.) - https://phabricator.wikimedia.org/T86849#980639 (10Aklapper) [22:13:47] 3MediaWiki-extensions-Flow, VisualEditor, Continuous-Integration: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#980649 (10hashar) Sorry, I swear I have though it got merged despite Matt -1. I must be tired :-] [22:19:23] 3Phabricator, Collaboration-Team: Trello migration script - https://phabricator.wikimedia.org/T821#980662 (10Spage) [22:25:09] greg-g: James_F: https://code.google.com/p/chromium/issues/detail?id=449299 [22:25:13] FYI :) [22:25:30] Krinkle: Fun. [22:26:22] 3Deployment-Systems: [scap] Log directly to logstash via redis input - https://phabricator.wikimedia.org/T86969#980687 (10bd808) Scap already format it's messages for logstash so the change needed is to [[https://github.com/Seldaek/monolog/blob/master/src/Monolog/Handler/RedisHandler.php#L48|mimic the Monolog Re... [22:27:09] 3Phabricator, Collaboration-Team: Trello migration script - https://phabricator.wikimedia.org/T821#980691 (10Spage) [22:36:43] 3Phabricator: Mingle migration script - https://phabricator.wikimedia.org/T822#980742 (10Eloquence) Yes, the Mingle instance is free to us. I'll ping if there's any change in that relationship. [22:41:28] marxarelli|lunch: are you around? I have a strange error in a newly installed ruby setup for browser tests for Elena [22:43:00] marxarelli: forget it, bundle update fixed it [22:43:49] chrismcmahon: alrighty [22:47:49] (03PS2) 10Krinkle: mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) [22:53:19] marxarelli: Elena can run Cucumber tests for beta labs on her laptop now [22:54:43] chrismcmahon: great! [23:07:59] (03CR) 10Krinkle: [C: 032] mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) (owner: 10Krinkle) [23:08:03] (03PS3) 10Krinkle: mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) [23:08:09] (03CR) 10Krinkle: [C: 032] mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) (owner: 10Krinkle) [23:10:07] test [23:10:53] That seems undesirable [23:11:00] https://phabricator.wikimedia.org/T86864#978342 [23:15:28] (03Merged) 10jenkins-bot: mediawiki-core-jsduck-publish: Add trailing slash in rsync commmand [integration/config] - 10https://gerrit.wikimedia.org/r/185267 (https://phabricator.wikimedia.org/T86864) (owner: 10Krinkle)