[00:02:48] 05Gitblit-Deprecate: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2370301 (10Danny_B) [00:05:00] Project beta-update-databases-eqiad build #9059: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9059/ [00:06:53] thcipriani: hey, can you +2 this? https://gerrit.wikimedia.org/r/#/c/293419/ [00:07:16] I want the prod serice working in the beta cluster to see if it's working properly [00:09:43] Amir1: will you be around for a bit? SWAT is still ongoing from the looks of it, I don't want to merge something while that's happening (it's confusing for the deployer). I'll merge it post-SWAT, post-folks deploying things. [00:11:05] thcipriani: depends, I'm in airport right now. Can you tell how much I need to wait? [00:13:44] unclear, looks like the last patch is going out now. [00:15:07] thcipriani: okay. I might be off, but the patch is super trivial and doesn't require my supervision. so please merge when you think is okay :) [00:15:35] Amir1: sure. [00:15:44] thanks :) [00:16:58] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #44: 04FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [00:25:04] 06Release-Engineering-Team, 15User-greg: RelEng 2015/16 Annual Reviews - https://phabricator.wikimedia.org/T136434#2370333 (10greg) Removing the goals as a blocker; still refining with people but done with their reviews. [00:25:12] 06Release-Engineering-Team, 15User-greg: RelEng 2015/16 Annual Reviews - https://phabricator.wikimedia.org/T136434#2370334 (10greg) [00:25:14] 06Release-Engineering-Team, 15User-greg: Create FY1617 annual personal goals (for RelEng team members) - https://phabricator.wikimedia.org/T134517#2370335 (10greg) [00:25:17] 10Deployment-Systems, 03Scap3, 07WorkType-NewFunctionality: Scap3 submodule space issues - https://phabricator.wikimedia.org/T137124#2370336 (10Ladsgroup) Adding @akosiaris who had the same concerns. [00:25:30] 06Release-Engineering-Team, 15User-greg: RelEng 2015/16 Annual Reviews - https://phabricator.wikimedia.org/T136434#2334978 (10greg) 05Open>03Resolved [00:28:01] 06Release-Engineering-Team, 15User-greg: Call out Community Tech dependency on RelEng for q1 - https://phabricator.wikimedia.org/T137336#2370340 (10greg) p:05Triage>03Normal a:05greg>03None [00:28:37] Amir1: merged. Should sync whenever then next beta-scap-eqiad runs. [00:29:02] awesome, thanks thcipriani ! [00:29:47] 06Release-Engineering-Team, 15User-greg: Call out Community Tech dependency on RelEng for q1 - https://phabricator.wikimedia.org/T137336#2365426 (10greg) This task came from a conversation during a 1:1 with @mmodell about, I believe, CommTech's goal for Q1 to build something on top of Phabricator (T133252? rig... [00:39:04] 06Release-Engineering-Team, 13Patch-For-Review, 15User-greg: Make a table of access levels per service RelEng maintains per person - https://phabricator.wikimedia.org/T135187#2370377 (10greg) I should migrate that here: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Access_list [00:45:53] 06Release-Engineering-Team, 15User-greg: Fix up RelEng related IRC notifications from Phabricator - https://phabricator.wikimedia.org/T137494#2370389 (10greg) [00:47:58] 06Release-Engineering-Team, 15User-greg: Update skillmatrix around start of Q1 - https://phabricator.wikimedia.org/T137511#2370393 (10greg) [01:03:20] 06Release-Engineering-Team, 15User-greg: Improve per-person assigned list - https://phabricator.wikimedia.org/T137513#2370433 (10greg) [01:05:00] Project beta-update-databases-eqiad build #9060: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9060/ [01:20:05] Project beta-update-databases-eqiad build #9061: 04FAILURE in 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9061/ [01:26:36] 06Release-Engineering-Team, 15User-greg: Call out Community Tech dependency on RelEng for q1 - https://phabricator.wikimedia.org/T137336#2370456 (10bd808) I don't expect a major need from #release-engineering-team in Q1. The Striker code to create Diffusion repositories is complete now (or complete enough for... [02:16:31] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #49: 04FAILURE in 3 min 31 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/49/ [02:54:36] (03PS1) 1020after4: Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 [02:54:38] (03PS1) 1020after4: fix puppet-lint warnings. [integration/config] - 10https://gerrit.wikimedia.org/r/293672 [03:05:00] Project beta-update-databases-eqiad build #9062: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9062/ [04:05:00] Project beta-update-databases-eqiad build #9063: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9063/ [04:43:33] 06Release-Engineering-Team, 15User-greg: Call out Community Tech dependency on RelEng for q1 - https://phabricator.wikimedia.org/T137336#2370569 (10greg) 05Open>03Resolved a:03greg Awesome, thanks. Not calling it out for next quarter then explicitly. [04:45:16] ACKNOWLEDGEMENT - jenkins_zmq_publisher on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 8888: Connection refused Muehlenhoff Setup ongoing [05:05:00] Project beta-update-databases-eqiad build #9064: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9064/ [06:05:00] Project beta-update-databases-eqiad build #9065: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9065/ [06:16:07] (03PS15) 10Lethexie: Add the SpaceBeforeControlStructureBraceSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275747 (https://phabricator.wikimedia.org/T130004) [07:05:00] Project beta-update-databases-eqiad build #9066: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9066/ [07:52:25] 05Continuous-Integration-Scaling, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2370750 (10hashar) Zuul (since 2.1.0-95) now measures the time for a build to actually start on a Node. That represents how... [07:57:16] 06Release-Engineering-Team, 15User-greg: Improve per-person assigned list - https://phabricator.wikimedia.org/T137513#2370433 (10hashar) That is quite nice. I now have learned you can include a workboard on a task and that the workboard search can uses tabs!! Maybe we could use that during our 1/1 or weekly... [07:59:30] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2370770 (10hashar) Indeed I had to revert Zuul 2.1.0-151 shortly after deploying it. It stopped receiving/processing events after a few minutes which I haven't managed to... [08:00:05] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/292346 (https://phabricator.wikimedia.org/T136610) (owner: 10Hashar) [08:04:21] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2370793 (10Paladox) @hashar could that be because of the new connections. [08:05:01] Project beta-update-databases-eqiad build #9067: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9067/ [08:08:09] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370809 (10hashar) [08:08:19] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370824 (10hashar) [08:08:21] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2120489 (10hashar) [08:08:33] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370809 (10hashar) [08:08:35] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2370825 (10hashar) [08:08:43] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2340878 (10hashar) 05Open>03stalled p:05Triage>03Normal a:05hashar>03None [08:08:52] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2120489 (10hashar) 05Open>03stalled p:05Triage>03Normal [08:12:37] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370838 (10Paladox) @hashar could it be we have to do http://docs.openstack.org/infra/zuul/connections.html [08:14:09] 10Continuous-Integration-Infrastructure, 10Wikidata, 05MW-1.28-release-notes, 13Patch-For-Review, and 2 others: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188#2370840 (10hashar) With T136301 the qunit jobs have been migrated to Nodepool instances with Debia... [08:15:11] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2370842 (10hashar) [08:15:30] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363409 (10hashar) Pending deployment. [08:17:45] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370847 (10hashar) [08:18:34] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2370809 (10hashar) @Paladox Yup that is what 66c8e52 to 30a433b is mostly about: switching to that new connection system. Something is broken t... [08:23:45] (03Abandoned) 10Hashar: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) (owner: 10Hashar) [08:24:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2370856 (10hashar) 05Open>03declined T137358 is declined, pending outcome of an archite... [08:42:18] (03CR) 10Hashar: "This change replace the old jobs entirely. The new ones are not known to pass on Nodepool instances and there is no straightforward way to" [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) (owner: 10Hashar) [08:59:22] 10Browser-Tests-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: migrate mwext-mw-selenium to Nodepool instances - https://phabricator.wikimedia.org/T137112#2370971 (10hashar) https://gerrit.wikimedia.org/r/293146 is merged providing libav-tools I had the Jessie image refreshed yesterda... [09:05:00] Project beta-update-databases-eqiad build #9068: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9068/ [09:06:54] 10Browser-Tests-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: migrate mwext-mw-selenium to Nodepool instances - https://phabricator.wikimedia.org/T137112#2370999 (10hashar) I have confirmed the `mwext-mw-selenium-jessie` job got past the missing `avconv`. [09:50:19] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371024 (10Paladox) @hashar but doint we need to update the zuul.conf file with those new configs for connections. [10:04:41] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371051 (10Paladox) @hashar comparing https://phabricator.wikimedia.org/rCIZU4b0d435342c0bd3001439746a84c8b470534c8c9 And https://phabricator.wiki... [10:05:00] Project beta-update-databases-eqiad build #9069: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9069/ [10:06:07] (03PS1) 10Gergő Tisza: Make MultimediaViewer browser tests non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/293699 [10:06:29] hashar: is ^ the correct syntax? [10:07:01] (03CR) 10jenkins-bot: [V: 04-1] Make MultimediaViewer browser tests non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/293699 (owner: 10Gergő Tisza) [10:07:18] apparently not. [10:08:19] tgr: :) [10:08:39] tgr: yeah that template does not exist [10:08:58] I thought the regex at line 596 makes that work for all templates [10:09:09] tgr: I guess the tests got added to make sure the browser tests are all green, if they suddenly fails that should indicate there is a trouble in the code [10:09:17] or the browser tests havent been kept in sync with the code somehow [10:09:18] then [10:09:35] if there is no interest in keeping it voting, I would just drop the job entirely [10:10:02] we also have a set of jobs that run on a daily basis at https://integration.wikimedia.org/ci/view/Selenium/job/selenium-MultimediaViewer/ [10:10:07] zeljkof: ^ [10:11:01] it would be nice if someone took the effort to fix those tests, but I definitely won't have time for a while, and they block simple bugfixes [10:11:14] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371089 (10Paladox) https://github.com/openstack-infra/zuul/blob/352264b3c22b501ae53c1643e31df2ccdfc89388/tests/fixtures/layouts/good_connections1... [10:11:26] MV gets so few updates, chances are those tests have been broken for months [10:12:05] at least it passed on May 26th [10:12:23] from the last merged change that got the job green https://gerrit.wikimedia.org/r/#/c/290845/ [10:12:48] well I am fairly sure https://gerrit.wikimedia.org/r/#/c/293630/ is not broken by the change in the commit [10:14:12] probably browser tests that are voting should run daily as well, they are easily broken by changes to other repos [10:15:13] anyway if you prefer not having non-voting jobs, I'll just wait until someone fixes them, my patches are not urgent [10:15:27] unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055) (Selenium::WebDriver::Error::WebDriverError) [10:15:28] bah [10:15:37] tgr: yeah that looks like an issue on the CI boxes for some reason :( [10:17:13] firefox got upgraded from 46 to 47 [10:17:28] zeljkof: firefox got upgraded on Trusty boxes, and maybe the Selenium webdriver needs some update :( [10:18:50] hashar: uh oh [10:19:04] I do not think there is new version of selenium [10:19:09] let me check [10:19:17] tgr has a couple patches for MultimediaViewer that are entirely unrelated to selenium [10:19:24] but has mwext-mw-selenium failing [10:19:36] https://integration.wikimedia.org/ci/job/mwext-mw-selenium/7159/console [10:19:43] https://integration.wikimedia.org/ci/job/mwext-mw-selenium/7160/console [10:19:48] the .mp4 video is quite useless [10:19:59] shows up as a black screen since Selenium never managed to acquire a browser [10:20:51] I am wrong [10:20:51] https://rubygems.org/gems/selenium-webdriver/ [10:21:02] 2.53.1 - June 9, 2016 (877 KB) [10:21:05] released yesterday [10:21:22] which commit is that? [10:21:38] https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/MultimediaViewer,n,z [10:21:41] top two ones [10:21:42] hashar, tgr ^ [10:21:42] which commit is failing? [10:21:47] ok, checking [10:21:52] the commit are entirely harmless [10:22:49] 2.53.1 is not even tagged in upstream source repo ... [10:23:06] (03CR) 10Zfilipin: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/293699 (owner: 10Gergő Tisza) [10:23:25] and for MultimediaViewer the Gemfile.lock has selenium-webdriver (2.53.0) [10:23:28] in unrelated news, anyone familiar with our logging config? I need a review for https://gerrit.wikimedia.org/r/293701 so I can unbreak AuthManager metrics and the usual suspects are still asleep [10:24:04] tgr: I am not unfortunately :( [10:24:15] tgr: you could try beta first and see what happens there? [10:25:33] I guess the reasonable thing to do is to is to just rename back the channels instead of messing with the config in an emergency deploy [10:26:23] zeljkof: doing colordiff -u -r selenium-webdriver-{2.53.0,2.53.1}|less -R [10:26:31] +Firefox: [10:26:31] + * Support for latest geckodriver (formerly wires) [10:26:31] + * Escape selector when converting to CSS (issue 2235) [10:26:36] so maybe 2.53.1 fix it [10:26:52] hashar: yes, testing it now [10:27:03] https://gerrit.wikimedia.org/r/#/c/293702/ [10:27:18] or not [10:27:21] (03CR) 10Gergő Tisza: "Hashar explained this is a bad idea, feel free to abandon it." [integration/config] - 10https://gerrit.wikimedia.org/r/293699 (owner: 10Gergő Tisza) [10:27:37] zeljkof: that is the same as tgr patches really [10:27:42] his patches are harmless [10:29:09] zeljkof: bumped the webdriver with https://gerrit.wikimedia.org/r/293703 [10:30:34] https://github.com/SeleniumHQ/selenium/commit/3c0e38c54d9c3275c3fda0c6d0a4e961aa806c23 [10:31:20] the bump does not solve it :( [10:33:19] [firefox] [10:33:19] bah [10:42:33] hashar: sorry, got distracted [10:42:41] will check what is going on [10:44:16] can we downgrade firefox? [10:44:24] an alternative is to use chrome [10:44:43] still checking if there is a way to fix firefox [10:47:32] can you reproduce locally ? [10:47:45] one of the trouble of mwext-mw-selenium and selenium jobs in general is the lack of browser logs :( [10:47:54] we can surely pin firefox back to v46 [10:48:02] did that for chromium a couple weeks ago [10:48:08] since chromium v50 was faulty [10:52:47] let me try running the tests locally [10:52:57] I have ff 47 [10:53:08] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371194 (10hashar) It is back compatible with the old way of defining connections (such as a `[gerrit]` section) https://github.com/openstack-infr... [10:54:06] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371198 (10hashar) [10:55:06] hashar: hm, the tests are failing locally too, but I can see firefox opening, and the error message is [10:55:11] end of file reached (EOFError) [10:56:06] but ff is crashing [10:56:30] please revert ff to 46 [10:56:41] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371205 (10hashar) And I think P3204 is the thread dump, possibly deadlocked with Crypto.Random.UserFriendlyRNG {P3204} [10:58:03] * zeljkof is out of lunch [11:05:00] Project beta-update-databases-eqiad build #9070: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9070/ [11:10:27] zeljkof: https://github.com/SeleniumHQ/selenium/issues/1862 :D [11:10:45] worktype-new feature!! [11:11:13] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371233 (10Paladox) @hashar thanks for finding that. Do we update that package. [11:12:09] Looks like there is new firefox driver, marionette [11:12:31] and end of https://github.com/SeleniumHQ/selenium/issues/2110 as bunch of info [11:12:37] apparently there is also a bug in firefox [11:13:34] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver [11:13:56] and the ruby 2.53.1 is apparently just https://github.com/SeleniumHQ/selenium/commit/3c0e38c54d9c3275c3fda0c6d0a4e961aa806c23 [11:14:21] Will test in a few minutes [11:14:27] should mediawiki_selenium bump to marionette ? [11:15:11] and firefox release notes https://www.mozilla.org/en-US/firefox/47.0/releasenotes/ [11:15:18] guess now we will want a task in Phabricator :) [11:15:37] Fx 47 release notes state: [11:15:37] Selenium WebDriver may cause Firefox to crash on startup, use Marionette WebDriver instead [11:20:42] hashar: yes [11:20:58] will create a task and update mediawiki_selenium [11:21:44] zeljkof: then on the Marionette page at https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver it references a soft named geckodriver [11:21:49] which is apparently needed [11:21:55] and replace an utility previously named "wires" [11:21:57] I think it is bad naming [11:21:59] which we apparently do not rely on [11:22:07] it is all probably the same thing [11:22:08] with geckodriver being https://github.com/mozilla/geckodriver [11:22:23] so I am not sure how we drive firefox eventually :D [11:22:32] oh man [11:22:46] so there is selenium/webdriver, marionette and geckodriver/wires [11:23:01] anyways, will update mw_selenium and see [11:24:39] 10Browser-Tests-Infrastructure: Update mediawiki_selenium to use Marionette - https://phabricator.wikimedia.org/T137540#2371247 (10zeljkofilipin) [11:30:13] food time [11:30:18] then will write the gallium related doc [11:43:40] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371296 (10Paladox) Or do you know how to fix it please. Or could it be a bug in zuul. [11:58:20] * zeljkof is out to lunch, this time for real [12:05:01] Project beta-update-databases-eqiad build #9071: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9071/ [12:09:19] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2371364 (10Paladox) @hashar could it be that gallium was the problem since it went faulty on wednesday but you turned this update a week before. S... [12:09:20] (03PS2) 10Jean-Frédéric: Add npm template to labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/293640 (https://phabricator.wikimedia.org/T137544) [12:15:41] (03PS3) 10Jean-Frédéric: Add npm template to labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/293640 (https://phabricator.wikimedia.org/T137544) [12:22:52] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #45: 04FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/45/ [12:25:24] hashar: It seems castor is down. [12:25:34] both fetching and saving is exiting with code 1. [12:25:35] in test pipeline [12:25:50] Or maybe it's not supposed to run on test pipeline? [12:26:03] bottom of https://integration.wikimedia.org/ci/job/npm-node-4.3/16075/console [12:26:27] see https://integration.wikimedia.org/ci/job/castor-save/88782/console [12:26:41] only runs on gate-and-submit and build being a success [12:26:55] the failure is definitely distracting though [12:42:22] hashar: https://twitter.com/samccone/status/739166801427210240 [12:42:30] "Headless Chrome is coming so soon" [12:43:58] (03PS4) 10Jean-Frédéric: Add npm template to labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/293640 (https://phabricator.wikimedia.org/T137544) [12:55:10] hashar: please downgrade ff for now [12:55:18] cant do [12:55:28] I found a way to fix selenium, but can not do it today [12:55:41] I have to go early, will work tomorrow [12:55:57] you can't downgrade ff? [12:57:22] * zeljkof has to go [12:58:45] zeljkof: going to take a couple hours which I cant commit to [12:58:58] need to grab the old firefox .deb from Ubuntu upstream [12:59:00] push them to the slaves [12:59:06] and pin firefox in puppet [13:00:02] we did that a couple weeks ago for chromium with https://phabricator.wikimedia.org/T136188#2332093 [13:05:00] Project beta-update-databases-eqiad build #9072: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9072/ [14:05:01] Project beta-update-databases-eqiad build #9073: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9073/ [14:34:28] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2371835 (10hashar) [14:37:11] (03PS1) 10Hashar: zuul: make mwext-mw-selenium non voting [integration/config] - 10https://gerrit.wikimedia.org/r/293732 (https://phabricator.wikimedia.org/T137561) [14:38:06] (03CR) 10Hashar: [C: 032] zuul: make mwext-mw-selenium non voting [integration/config] - 10https://gerrit.wikimedia.org/r/293732 (https://phabricator.wikimedia.org/T137561) (owner: 10Hashar) [14:38:26] zeljkof: made them non voting for now [14:38:30] all I can afford / do safely [14:39:09] (03Merged) 10jenkins-bot: zuul: make mwext-mw-selenium non voting [integration/config] - 10https://gerrit.wikimedia.org/r/293732 (https://phabricator.wikimedia.org/T137561) (owner: 10Hashar) [14:39:41] be back later [14:57:37] hasharAway i think zuul problems could be with https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Gearman_deadlock [14:57:45] Meaning the new version we tryed [15:05:00] Project beta-update-databases-eqiad build #9074: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9074/ [15:14:40] (03PS1) 10Paladox: zuul: sync with upstream @057aed1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/293741 [15:14:54] hasharAway ^^ [15:15:38] !log T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/39/293739/1 && git cherry-pick FETCH_HEAD [15:15:39] T137561: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561 [15:15:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:17:42] (03PS2) 10Paladox: zuul: sync with upstream @057aed1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/293741 [15:18:31] !log T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && wget 'https://ubuntu.wikimedia.org/ubuntu/pool/main/f/firefox/firefox_46.0.1%2bbuild1-0ubuntu0.14.04.3_amd64.deb'" [15:18:32] T137561: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561 [15:18:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:19:54] !log T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && dpkg -i firefox_46.0.1+build1-0ubuntu0.14.04.3_amd64.deb" [15:19:55] T137561: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561 [15:19:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:28:39] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 13Patch-For-Review: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2371835 (10JanZerebecki) That didn't work, see https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer/3504/consoleFull , same sym... [15:29:34] !log T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git reset --hard 1e1ff12b13b73b5c5e2015a72f51561f10b305d0 [15:29:34] T137561: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561 [15:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:31:25] mediawiki-extensions-php55 seems to have suddenly started failing, but I don't see anything changed recently to explain it. See https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/4849/console for example. [15:36:00] (03PS1) 10JanZerebecki: Revert "zuul: make mwext-mw-selenium non voting" [integration/config] - 10https://gerrit.wikimedia.org/r/293744 [15:36:17] hmmm, in that specific instance, it looks like all the
are off by one which is strange. [15:37:20] (03PS2) 10JanZerebecki: Revert "zuul: make mwext-mw-selenium non voting" [integration/config] - 10https://gerrit.wikimedia.org/r/293744 [15:39:09] Yippee, build fixed! [15:39:10] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #42: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/42/ [15:43:38] (03CR) 10JanZerebecki: [C: 032] Revert "zuul: make mwext-mw-selenium non voting" [integration/config] - 10https://gerrit.wikimedia.org/r/293744 (owner: 10JanZerebecki) [15:44:38] (03Merged) 10jenkins-bot: Revert "zuul: make mwext-mw-selenium non voting" [integration/config] - 10https://gerrit.wikimedia.org/r/293744 (owner: 10JanZerebecki) [15:44:48] jzerebecki: Hi could you review https://gerrit.wikimedia.org/r/#/c/293741/ please. [15:45:47] We should add the extensions in a release as submodules of core. [15:46:06] Would ease bundling maintenance imho [15:46:18] Yippee, build fixed! [15:46:19] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #42: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/42/ [15:46:19] Would also allow them to be part of the tag. [15:46:28] I like this idea [15:46:31] * ostriches files task [15:49:14] !log reloading zuul for 8c048fb..272d1ec [15:49:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:51:34] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 13Patch-For-Review: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2371982 (10JanZerebecki) reverted https://gerrit.wikimedia.org/r/293732 zuul: make mwext-mw-selenium non voting in https://gerrit.wikimedia.or... [15:51:48] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2371983 (10JanZerebecki) [15:56:10] 10MediaWiki-Releasing, 06Release-Engineering-Team: Include release extensions/skins as submodules of core (maybe vendor too?) - https://phabricator.wikimedia.org/T137564#2371986 (10demon) [16:05:00] Project beta-update-databases-eqiad build #9075: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9075/ [16:13:34] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2372095 (10hashar) +2 about all what @JanZerebecki did ! That is how we handled Chromium pinning to v49. Kudos Jan! [16:23:04] (03CR) 10Hashar: [C: 031] Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 (owner: 1020after4) [16:23:27] (03CR) 10Hashar: [C: 032] fix puppet-lint warnings. [integration/config] - 10https://gerrit.wikimedia.org/r/293672 (owner: 1020after4) [16:24:19] (03CR) 10Paladox: [C: 031] Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 (owner: 1020after4) [16:43:39] (03CR) 1020after4: [C: 032] Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 (owner: 1020after4) [16:44:30] (03CR) 10jenkins-bot: [V: 04-1] fix puppet-lint warnings. [integration/config] - 10https://gerrit.wikimedia.org/r/293672 (owner: 1020after4) [16:45:11] (03PS2) 1020after4: Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 [16:45:37] (03CR) 1020after4: [V: 032] Install arcanist on contintcloud slaves [integration/config] - 10https://gerrit.wikimedia.org/r/293671 (owner: 1020after4) [16:46:16] (03CR) 1020after4: [C: 032 V: 032] fix puppet-lint warnings. [integration/config] - 10https://gerrit.wikimedia.org/r/293672 (owner: 1020after4) [16:46:23] (03PS2) 1020after4: fix puppet-lint warnings. [integration/config] - 10https://gerrit.wikimedia.org/r/293672 [17:05:00] Project beta-update-databases-eqiad build #9076: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9076/ [17:08:28] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2372357 (10hashar) If I knew the cause or a commit that fix it, that would have been done already. This task is about investigating the issue whic... [17:11:18] (03PS4) 10Cdentinger: Make DonationInterface test run against 1_26 branch. [integration/config] - 10https://gerrit.wikimedia.org/r/293344 (https://phabricator.wikimedia.org/T137213) [17:28:39] (03CR) 10Paladox: [C: 031] Make DonationInterface test run against 1_26 branch. [integration/config] - 10https://gerrit.wikimedia.org/r/293344 (https://phabricator.wikimedia.org/T137213) (owner: 10Cdentinger) [17:46:28] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Danny_B) First shot just off-head, untested. HTH. ``` # replace all %2f in repo name with / ^/(commit|log... [18:05:01] Project beta-update-databases-eqiad build #9077: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9077/ [18:12:22] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2372568 (10Dzahn) Thank you DannyB! Btw, we have setup a labs instance for testing http://git.wmflabs.org/ and hap... [18:25:14] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2372576 (10demon) Also needs to handle: ``` git.wikimedia.org/ -> phabricator.wikimedia.org/diffusion/query/active/... [18:30:59] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2372602 (10Paladox) [18:31:17] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Paladox) [18:34:34] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2372604 (10Paladox) [18:44:54] 05Gerrit-Migration, 10Differential, 13Patch-For-Review: Set differential.always-allow-close to true - https://phabricator.wikimedia.org/T131623#2372641 (10mmodell) 05Open>03declined [18:55:03] reeehh [18:55:43] twentyafterfour: did you get arcanist installed on nodepool images and updated them ? :D [18:56:38] hashar: I don't have access to loadGitBranchPositions [18:56:40] er [18:56:49] to labnodepool1001.eqiad.wmnet [18:57:02] lol bad paste no doughnut [18:57:06] hmm [18:57:24] arent you in contint-admins group ? [18:57:40] you are [18:58:27] twentyafterfour: you should be able to login though. At least puppet wise [18:58:33] and tyler definitely managed to connect as well [18:58:45] via contint-admin groups from modules/admin/data/data.yaml [18:59:06] applied on the host via hieradata/hosts/labnodepool1001.yaml [19:00:02] hmm... [19:00:33] maybe a ssh host routing issue ? [19:00:33] FWIW, can confirm that I'm able to get in (just checked) [19:00:36] now it lets me in. I guess I was doing it wrong before [19:00:44] where labnodepool1001.eqiad.wmnet would not be routed via a bastxxxx? [19:01:42] and I wrote a rough doc about gallium phase out https://docs.google.com/a/wikimedia.org/document/d/1FR6IOP_4rxHRhLesYuokB-AhUzKBMGE4PEoOz2Wpk3A/edit?usp=sharing [19:02:06] with a lame overview for everything on labs https://docs.google.com/a/wikimedia.org/drawings/d/1E7dEx--qRHku9Kr8QVrd4emj9RHltZ6Rg7jURjgvpMk/edit?usp=sharing [19:02:40] I wish I could insert the drawing in the doc [19:05:00] Project beta-update-databases-eqiad build #9078: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9078/ [19:05:21] and my kid is reviewing the drawing with me :D [19:06:36] :) [19:06:57] she has concerns about the colors I have picked [19:09:37] ok running the update process [19:10:04] neat [19:10:13] you should see puppet playing with it [19:10:20] 2016-06-10 19:10:07,968 INFO paramiko.transport: Connected (version 2.0, client OpenSSH_6.7p1) [19:10:22] 2016-06-10 19:10:08,110 INFO paramiko.transport: Authentication (publickey) failed. [19:10:23] then at the end something like: ./snapshot completed [19:10:24] 2016-06-10 19:10:08,279 INFO paramiko.transport: Authentication (publickey) failed. [19:10:25] yeah [19:10:33] those auth things are nodepool being lame [19:11:02] it basically tries to ssh 5 times with usernames of cloud init images [19:11:16] so would try root , cloudinit, redhatcloud, ubuntucloud, debian [19:11:19] something like that [19:11:38] ideally the username would be filled in Nodepool config file, but there is no support for that [19:12:16] the ssh key is provided via openstack and injected via cloud init [19:12:34] once it managed to ssh to it, it would run the setup_node.sh script which mainly puppet apply the ciimage.pp [19:12:42] at the end it invokes sync [19:12:49] /Stage[main]/Phabricator::Arcanist/Git::Clone[phabricator/arcanist]/File[/usr/local/share/arcanist]/ensure: created [19:12:52] then ask openstack to freeze the instance [19:12:54] neat!!!! [19:14:07] Syncing filesystem [19:14:08] 2016-06-10 19:13:34,860 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wikimedia: ./setup_node.sh complete (hostname: ci-jessie-wikimedia-1465585715) [19:14:23] is that the end ^ then? [19:15:15] mostly [19:15:21] then there are the openstack api call [19:15:41] it would iterate api calls until the instance has been snapshotted [19:15:42] it's just repeating 2016-06-10 19:15:02,787 INFO urllib3.connectionpool: Starting new HTTP connection (1): labnet1002.eqiad.wmnet [19:15:46] yeah [19:15:50] that is the lame loop [19:16:01] it keeps asking for the instance metadata [19:16:08] ^ that behavior worries me about nodepool [19:16:09] which has some state like "SNAPSHOTTING" [19:16:52] thcipriani: the api spam ? [19:17:23] after a while it should have the snapshot done [19:17:38] it's certainly not very user friendly UI [19:17:42] yeah, it was mentioned that it looked like nodepool was hammering openstack from time-to-time [19:18:04] and in the output of some commands it does seem that way [19:18:37] especially when the timeout is nominally 60 seconds, it doesn't seem to be waiting for any kind of timeout afaict [19:18:43] rapidly polling in a loop to monitor the status seems bad but even worse it's not giving me any useful status info in the output [19:19:58] yeah --debug might gives more [19:20:08] Project beta-update-databases-eqiad build #9079: 04STILL FAILING in 7.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9079/ [19:20:32] still going [19:24:25] bah [19:24:34] still not complete ? [19:25:00] usually it takes a couple minutes [19:26:46] twentyafterfour: I guess you can kill / cancel it [19:26:49] and try again :( [19:27:36] hashar: I can't cancel it [19:27:41] ctl-c doesn't do the trick anyway [19:28:11] ctl-z + kill did the trick [19:28:23] bah [19:28:38] guess they forgot to handle SIGINT [19:28:53] i killed it [19:28:53] trying again [19:28:58] twentyafterfour: pass --debug [19:29:01] ok [19:29:03] nodepool --debug ... [19:29:09] would spurt a few more info potentially [19:29:32] we have the nova-api log [19:29:40] but lack the compute / glance logs :D [19:29:49] so hard to know what is really going on [19:30:16] Forbidden: Quota exceeded for cores,instances: Requested 2, but already used 40 of 40 cores (HTTP 403) [19:30:17] eek [19:30:24] legoktm could i update https://www.mediawiki.org/wiki/Git/Reviewers#mediawiki.2F.2A with extension.json and skin.json [19:30:26] please [19:31:12] hashar https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm-trusty/2020/ has been running for 2+ hrsd [19:31:13] hrs [19:31:37] It says this 17:48:44 Build timed out (after 30 minutes). Marking the build as failed. [19:31:43] but then it dosent stop the build [19:31:46] wierd [19:32:15] paladox: need a threaddump in jenkins i guess [19:32:28] hashar oh [19:37:03] hashar: so it failed? [19:37:43] twentyafterfour: there is a limited quota [19:38:03] and maybe when one ask the snapshot when the quota is exceeded that has a side effect of some sort? :( [19:38:05] yes isn't nodepool supposed to keep us below that quota? [19:38:05] no idea really [19:38:25] the second run seemed like it might have been successful? [19:38:54] you should get a message at the end stating it is complete [19:39:43] There are no nodes with the label ‘ci-jessie-wikimedia [19:40:28] according to jenkins [19:41:12] paladox: the instance is borked somehow ... No idea what happens on it [19:42:52] twentyafterfour: it build apparently [19:42:57] twentyafterfour: nodepool image-list [19:43:00] list the snapshots [19:43:13] id 776 is the refreshed one [19:43:25] state: ready | age: 0.13 [19:43:40] so from now on, instances booted from that snapshot should have arcanist [19:45:11] to find which image instances are running I usually head to horizon https://horizon.wikimedia.org/project/instances/ [19:46:11] well it succeeded apparently. nodepool slaves have arcanist now [19:46:17] kudos ! [19:46:45] it is really awkward process [19:46:50] cool so now my jenkins job works [19:51:07] so does that work without the api-key ? [19:51:23] from yesterday, I would assume the Jenkins Phabricator plugin to handle that [19:51:46] I mean, if arcanist is present , the plugin magically manage to fetch the diff with some tokens dynamically passed [19:52:33] twentyafterfour: also have you cleaned up the hack that used castor to inject arcanist ? [19:52:48] this works https://git.wmflabs.org/log/mediawiki%2Fcore.git/refs%2Fheads%2Fmaster [19:52:49] :) [19:53:06] :D [19:53:34] and https://git.wmflabs.org/tree/mediawiki%2Fcore.git/refs%2Fheads%2Fmaster [19:53:41] Danny_B got it working ^^ [19:53:42] paladox: operations have some kind of script to test the MediaWiki redirects. Cant remember where it is though [19:53:59] hashar oh we set up an instance on labs to test [19:54:02] https://git.wmflabs.org/ [19:54:07] I have access to it [19:54:09] yeah I noticed [19:54:17] but how do you tests the redirects? Manually? [19:54:24] Yep. and yes manually. [19:54:34] We want the public to test too. [19:58:12] hashar i doint think we can do /metrics/ [19:58:23] since there is no metrics or anything similar to redirect too. [19:58:28] in phabricator [19:58:30] https://phabricator.wikimedia.org/T137224#2372576 [19:59:21] paladox: I was speaking with mutante yesterday about that [19:59:31] hashar oh [19:59:38] we were wondering whether we have a list of actual requests being done on git.wikimedia.org [19:59:45] yep he said about that. [19:59:49] since it is behind Varnish, we have the requests logged to an analytics cluster [19:59:59] but apparently there is only one day of data [20:00:14] hashar i think i found a website that tracks this https://www.similarweb.com/ [20:00:36] https://www.similarweb.com/website/git.wikimedia.org [20:03:15] hashar https://www.similarweb.com/website/git.wikimedia.org/summary?r=mediawiki/core.git [20:03:37] damnit. Your version of 'xhpast' is unbuilt or out of date. Run this script to build it.: [20:03:39] 20:02:51 [20:03:41] 20:02:51 $ /usr/local/share/libphutil/scripts/build_xhpast.php [20:03:52] hashar: how can I manually run a command in the image before taking the snapshot? [20:03:57] I get that on windows [20:05:12] hashar on git.wikimedia.org viewing tree in mw core there is 31.19% traffic [20:05:56] It also shows the traffic for the other branch too at https://www.similarweb.com/website/git.wikimedia.org/summary?r=mediawiki/core.git#websiteContent [20:06:01] twentyafterfour: so arcanist depends on some binary/native code? :( [20:06:15] 2Fextensions%2FWikibase.git/master 4.28% [20:06:28] …git.wikimedia.org/tree/operations%2Fpuppet.git 4.28% [20:07:44] hashar [20:07:45] Git.wikimedia.org has received an estimated 40,484,000 visits over the last 30 days. [20:07:58] twentyafterfour ostriches ^^ [20:08:06] paladox: with almost all of that being crawlers [20:08:16] http://www.trafficestimate.com/git.wikimedia.org [20:08:19] yes maybe [20:08:22] which would get rid of the ones failing for sure [20:08:23] oh [20:08:31] dont trust trafficestimate for sure :D [20:08:46] who knows how they come to that number really [20:08:47] hashar: yes, it needs to compile xhpast [20:08:59] I guess I'm just gonna put the compiled binary in the git repo [20:09:09] Yes, probaly i shoulden [20:09:10] yawnnn [20:09:16] Im going through the list on google [20:09:24] twentyafterfour: I cant remember having to compile xhpast when I installed arcanist [20:09:47] hopefully one built on ubuntu vivid will work on jessie [20:09:59] hashar: it compiles automatically if it can [20:10:14] but the path is not writable on nodepool slaves [20:11:00] /usr/local/ yeah for sure [20:11:08] hashar is https://www.semrush.com/info/git.wikimedia.org?db=us this trustworthy [20:11:13] ideally it would fallback to the home dir ... [20:11:33] and I guess xhpast is a hard requirement isn't it ? [20:13:16] twentyafterfour: maybe we would want to get a .deb package for arcanist :D [20:16:52] twentyafterfour: https://packages.debian.org/sid/libphutil !!!!!!!!!!! [20:16:55] hashar https://gerrit.wikimedia.org/r/#/c/293741/ [20:17:15] twentyafterfour: though it does not have the binary :D [20:17:23] hashar: arcanist changes really often. packaging it may not be a good idea [20:17:30] well [20:17:43] if we ends up needing arcanist on prod I am pretty sure we would have to package it [20:17:46] it has to stay in sync with phabricator and it self-updates via git [20:17:51] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10MediaWiki-Unit-tests: mediawiki-extensions-qunit failing "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2372819 (10Legoktm) p:05Triage>03High I'm seeing this pretty often now... [20:17:59] or at least, I am sure shipping a binary via a git repo would be a no/no [20:18:03] hashar: we have it on prod deployed via puppet [20:18:08] ohh [20:18:20] I only just committed the binary :-/ [20:18:21] hashar: not watching football? France is playing ;) [20:18:27] so puppet would magically compile xhpast , then since it runs as root it lands on /usr/local/ ? [20:18:53] hashar: no it just has arcanist without xhpast [20:19:01] zeljkof: last time I heard of football was in 1998 when the second time the whole city was screaming I asked familly what was going on [20:19:12] zeljkof: France 2 - Brazil 0. And I went back to hacking :D [20:19:26] the xhpast bin doesn't go in /usr/local/bin [20:19:38] on Nodepool cant we do the same ? [20:20:13] it _needs_ xhpast to run arc lint [20:20:21] prod doesn't need to run arc lint [20:20:35] got it [20:20:57] hashar: I remember France kicking Croatia out of world championship in 98 2:1 :D [20:21:06] I guess I could put xhpast binary in puppet [20:21:07] maybe arc can be hinted at using a different prefix than /usr/local [20:21:25] also xhpast looks straightforward / plain simple and might be easy to package [20:21:47] yes it would be trivial to package but it has to be kept in sync with libphutil versions [20:21:54] so updating the package would be a pain [20:42:00] Yippee, build fixed! [20:42:00] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #51: 09FIXED in 59 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/51/ [21:05:00] Project beta-update-databases-eqiad build #9080: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9080/ [21:24:35] hashar legoktm twentyafterfour another failed build [21:24:35] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/554/console [21:24:39] 1+hr [21:24:51] jzerebecki ^^ [21:32:18] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2372974 (10Paladox) @demon I think we should redirect git.wikimedia.org to phabricator.wikimedia.org/diffusion/ [21:49:29] paladox: killed it [21:49:35] paladox: no clue what is happening there :/ [21:49:43] hashar ok thanks :) [21:49:54] I am heading to bed [21:50:00] waking up in 7 hours... :( [21:50:04] Ok [21:50:07] thanks for the gitblit redirects [21:50:09] Waking up in 7hrs [21:50:12] and the ci related comments! [21:50:18] Your welcome [21:50:24] Waking up in 7hrs? [21:50:42] hashar ^^ [21:58:17] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #50: 04FAILURE in 6 min 17 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/50/ [21:58:48] paladox: I wake up at 7am and it is midnight here :D [21:58:55] so yeah waking up in 7 hours from now [21:59:00] oh, thats early for a weekend [21:59:16] hashar ^^ [21:59:48] kids! [21:59:53] Oh ok [21:59:54] and when you get old you sleep less :) [21:59:59] oh [22:05:00] Project beta-update-databases-eqiad build #9081: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9081/ [22:57:26] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373135 (10Danny_B) >>! In T137224#2372974, @Paladox wrote: > @demon I think we should redirect git.wikimedia.org to... [22:58:03] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373136 (10Danny_B) a:03Danny_B [23:05:00] Project beta-update-databases-eqiad build #9082: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9082/ [23:05:28] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373153 (10Danny_B) [23:13:35] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373193 (10Danny_B) [23:40:13] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373229 (10Danny_B) [23:51:19] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373237 (10Paladox) [23:55:43] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373243 (10Paladox) [23:56:22] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Paladox) [23:58:11] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2373246 (10Paladox) [23:59:44] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Paladox)