[00:00:23] * marxarelli is distracted by long shadows breaching the hammock and a beer in the fridge [00:01:21] thcipriani: yay! `lsof -i :6379` shows established connections [00:01:43] marxarelli: hot diggity. [00:01:50] one from mira.deployment-prep and one from deployment-tin [00:02:01] nice, should be slaves [00:02:15] alright, lemme try a trebuchet deploy [00:02:55] thcipriani: yep yep. "slaveof 10.68.17.240" on mira [00:06:19] marxarelli: sigh, for whatever reason, remotes still think the upstream repo is on tin. [00:06:39] :/ [00:06:45] but that's a trebuchet thing, not a redis thing, so...that's good. [00:07:16] marxarelli: I'm going to go ahead and commit your change locally on puppetmaster [00:07:55] thcipriani: oh i just did [00:08:18] ah, cool, thanks [00:15:06] PROBLEM - Host deployment-bastion is DOWN: CRITICAL - Host Unreachable (10.68.16.58) [00:19:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:22:52] huh, well, you might think that trebuchet would attempt to rewrite the remote url if the deployment-server pillar value has changed, but you'd also, evidently, be wrong. [00:24:02] RECOVERY - Puppet failure on mira is OK: OK: Less than 1.00% above the threshold [0.0] [00:26:05] 23:00:30 [mediawiki-extensions-php55] $ /bin/bash -xe /tmp/hudson7204945152824979106.sh [00:26:06] 23:00:30 + /srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh [00:26:08] 23:00:30 ERROR 1269 (HY000) at line 1: Can't revoke all privileges for one or more of the requested users [00:26:11] At the end of https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/588/consoleFull [00:26:24] Also, apparently non-fatal notices halfway through: [00:26:34] 22:50:04 PHP Notice: Cannot find site jenkins_u2_mw in sites table [Called from Wikibase\Client\WikibaseClient::newSiteGroup in /mnt/jenkins-workspace/workspace/mediawiki-extensions-php55/src/extensions/Wikidata/extensions/Wikibase/client/includes/WikibaseClient.php at line 591] in /mnt/jenkins-workspace/workspace/mediawiki-extensions-php55/src/includes/debug/MWDebug.php on line 300 [00:26:41] (twice) [00:27:16] Going to try that one again since it was an hour and a half ago [00:27:50] I think I saw a ticket somewhere about adding new wikis to the sites table [00:27:56] when they get set up [00:33:51] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Special:CentralAutoLogin/checkLoggedIn redirects to wikimediafoundation.org on betalabs - https://phabricator.wikimedia.org/T126697#2021229 (10MaxSem) 3NEW [00:33:58] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:39:40] Ugh it broke again [00:39:42] * RoanKattouw files task [00:40:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Special:CentralAutoLogin/checkLoggedIn redirects to wikimediafoundation.org on Beta Cluster - https://phabricator.wikimedia.org/T126697#2021256 (10greg) [01:00:44] 10Continuous-Integration-Infrastructure: CI broken for Echo (possibly other things) with error "Can't revoke all privileges for one or more of the requested users" - https://phabricator.wikimedia.org/T126699#2021271 (10Catrope) 3NEW [01:34:46] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #532: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/532/ [02:51:05] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Special:CentralAutoLogin/checkLoggedIn redirects to wikimediafoundation.org on Beta Cluster - https://phabricator.wikimedia.org/T126697#2021433 (10Anomie) ``` $ curl -v 'http://login.wikimedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/chec... [02:54:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10Wikimedia-Apache-configuration, 6operations: Special:CentralAutoLogin/checkLoggedIn redirects to wikimediafoundation.org on Beta Cluster - https://phabricator.wikimedia.org/T126697#2021446 (10MaxSem) [02:57:00] 10Continuous-Integration-Infrastructure, 10IPSet, 5Patch-For-Review: IPSet::__construct() in gets into infinite loop when called from curl on a CI host - https://phabricator.wikimedia.org/T126495#2021453 (10PleaseStand) >>! In T126495#2019792, @BBlack wrote: > Circling back around to this and looking again:... [03:09:35] Project beta-scap-eqiad build #89564: 04FAILURE in 4 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89564/ [03:10:01] !log attempted to sync graphoid from gerrit 270166 from deployment-tin, but it wouldn't sync. Tried to git pull sca02, submodules wouldn't pull [03:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:40:36] Yippee, build fixed! [03:40:36] Project beta-scap-eqiad build #89567: 09FIXED in 5 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89567/ [04:31:30] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10Wikimedia-Apache-configuration, 6operations: Special:CentralAutoLogin/checkLoggedIn redirects to wikimediafoundation.org on Beta Cluster - https://phabricator.wikimedia.org/T126697#2021579 (10Tgr) Maybe during T124804 some redirect respon... [05:46:19] PROBLEM - Puppet failure on integration-slave-trusty-1002 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:09:50] Project beta-scap-eqiad build #89582: 04FAILURE in 5 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89582/ [06:26:22] RECOVERY - Puppet failure on integration-slave-trusty-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [06:38:20] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [06:40:08] Yippee, build fixed! [06:40:08] Project beta-scap-eqiad build #89585: 09FIXED in 5 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89585/ [07:13:26] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:09:46] 10Continuous-Integration-Config: Convert all MediaWiki extension phpunit jobs to use generic jobs - https://phabricator.wikimedia.org/T126682#2021711 (10hashar) I love the idea of a generic non voting job. Never thought about it! [08:21:17] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #857: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/857/ [08:31:19] https://integration.wikimedia.org/ci/job/mwext-testextension-php55/916/console uh what? [08:39:53] Project beta-scap-eqiad build #89597: 04FAILURE in 5 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89597/ [08:45:57] 08:16:37 + /srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh [08:45:57] 08:16:37 ERROR 1269 (HY000) at line 1: Can't revoke all privileges for one or more of the requested users [08:46:06] weird... [08:46:26] Nikerabbit: recheck? seems like a tmpfs issue [08:47:02] legoktm: ok that went away [08:52:42] (03CR) 10Hashar: [C: 031] "I told Paladox I would prefer that job to disappear but we are not there yet. That is a step toward phasing out Precise nodes so +1" [integration/config] - 10https://gerrit.wikimedia.org/r/270111 (owner: 10Paladox) [08:56:08] RECOVERY - Puppet failure on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [08:58:02] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:11:35] Yippee, build fixed! [09:11:35] Project beta-scap-eqiad build #89600: 09FIXED in 7 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89600/ [09:12:34] (03PS1) 10Legoktm: doc: Add HtmlFormatter [integration/docroot] - 10https://gerrit.wikimedia.org/r/270251 (https://phabricator.wikimedia.org/T125001) [09:13:56] (03CR) 10Legoktm: [C: 032] doc: Add HtmlFormatter [integration/docroot] - 10https://gerrit.wikimedia.org/r/270251 (https://phabricator.wikimedia.org/T125001) (owner: 10Legoktm) [09:14:30] (03Merged) 10jenkins-bot: doc: Add HtmlFormatter [integration/docroot] - 10https://gerrit.wikimedia.org/r/270251 (https://phabricator.wikimedia.org/T125001) (owner: 10Legoktm) [09:27:57] 10Continuous-Integration-Config: Have CI set `$wgScribuntoDefaultEngine = 'luasandbox` to speed up parser tests - https://phabricator.wikimedia.org/T126670#2021795 (10JanZerebecki) Sounds like we want to change it to do autodetection instead of adding special configuration to our CI. [09:53:05] (03CR) 10JanZerebecki: [C: 032] Update test for php55 [integration/config] - 10https://gerrit.wikimedia.org/r/270125 (owner: 10Paladox) [09:54:24] (03Merged) 10jenkins-bot: Update test for php55 [integration/config] - 10https://gerrit.wikimedia.org/r/270125 (owner: 10Paladox) [10:14:23] 10Browser-Tests-Infrastructure, 7JavaScript: Create a few tests using Nightwatch.js - https://phabricator.wikimedia.org/T126435#2021860 (10zeljkofilipin) I did not want to install Java on my machine, because it is a security nightmare. I have installed Ubuntu in Virtualbox virtual machine, installed java, node... [10:45:07] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 5Patch-For-Review, 3Wikipedia-Android-App: Wikipedia Android CI tests are failing - https://phabricator.wikimedia.org/T126532#2021915 (10hashar) [10:45:09] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 15User-greg: Migrate leftover tox jobs to CI Nodepool - https://phabricator.wikimedia.org/T126588#2021914 (10hashar) [10:46:13] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 5Patch-For-Review, 3Wikipedia-Android-App: Wikipedia Android CI tests are failing - https://phabricator.wikimedia.org/T126532#2016587 (10hashar) So the root cause was definitely Precise nodes not upgrading `setuptools` properly due to pupp... [10:54:24] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 15User-greg: Migrate leftover tox jobs to CI Nodepool - https://phabricator.wikimedia.org/T126588#2021925 (10hashar) [10:58:44] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-02: Support PHP 5.5 in CI for Wikidata stuff - https://phabricator.wikimedia.org/T126441#2021937 (10hashar) a:3JanZerebecki [10:59:39] 10Continuous-Integration-Config, 7Easy: translatewiki.net phplint job should use HHVM to lint (that is what prod is using) - https://phabricator.wikimedia.org/T97889#2021940 (10hashar) [11:09:58] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2021957 (10hashar) [11:12:17] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2021959 (10hashar) It is definitely an issue with the way Texlive is installed / set up on the CI Trusty slaves. I am wondering how that... [11:13:34] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2021963 (10hashar) [11:14:07] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance: Rebuild integration-dev (instance to build images) - https://phabricator.wikimedia.org/T126613#2021967 (10hashar) [11:14:09] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2008504 (10hashar) [11:14:24] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2008504 (10hashar) The instance I used to build image is gone (T126613) :-( [11:17:25] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2021976 (10hashar) @legoktm worked hard and committed to get that implemented and working properly. Today looks like everything is wo... [11:20:07] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974#2021980 (10hashar) [11:20:41] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974#16760 (10hashar) Removed the CI part, no need to be encumbered by that task until something is decided to act on which really depends on #mediawiki-releasing prioritization. [11:38:51] 10Continuous-Integration-Infrastructure: Remove PhantomJS from the CI infrastructure - https://phabricator.wikimedia.org/T113279#2022028 (10hashar) a:3Krinkle [11:39:00] 10Continuous-Integration-Infrastructure: Remove PhantomJS from the CI infrastructure - https://phabricator.wikimedia.org/T113279#2022029 (10hashar) 5Open>3Resolved Removed by @Krinkle with fc91d25b2f5d801b7302a24c22286a19d9c3c4cd on December 9th 2015. [11:45:26] 10Continuous-Integration-Infrastructure: Jenkins: Set up job in gate-and-submit to avoid submitting "DRAFT" or "WIP" commits - https://phabricator.wikimedia.org/T48860#2022040 (10hashar) There is a patch proposed upstream: * https://review.openstack.org/#/c/251373/ ** Filter events by commit message, Allow eve... [11:46:19] 10Continuous-Integration-Infrastructure: Zuul should not run jenkins-bot on changes for refs/meta/* - https://phabricator.wikimedia.org/T52389#2022043 (10hashar) [11:46:21] 10Continuous-Integration-Infrastructure: Jenkins: jenkins-bot reports spurious merge error when pushing changes to one of the gerrit config branches - https://phabricator.wikimedia.org/T66678#2022042 (10hashar) [11:50:48] (03PS2) 10Hashar: [FundraisingEmailUnsubscribe] Switch extension-jslint to jshint and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/269865 (owner: 10Paladox) [11:51:16] (03CR) 10Hashar: [C: 032] [FundraisingEmailUnsubscribe] Switch extension-jslint to jshint and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/269865 (owner: 10Paladox) [11:53:40] (03Merged) 10jenkins-bot: [FundraisingEmailUnsubscribe] Switch extension-jslint to jshint and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/269865 (owner: 10Paladox) [11:56:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: MySQL down on integration-slave-trusty-(1020|1021) - https://phabricator.wikimedia.org/T126615#2022060 (10hashar) Dont quote me, but I think that whenever the servers is hitting memory limit, the tmpfs misbehave and the database hosted o... [11:57:29] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: CI trusty slaves running out of memory - https://phabricator.wikimedia.org/T126545#2022062 (10hashar) [11:57:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: MySQL down on integration-slave-trusty-(1020|1021) - https://phabricator.wikimedia.org/T126615#2022061 (10hashar) [12:07:53] 10Continuous-Integration-Config: Timeouts in mediawiki-extensions-php53 - https://phabricator.wikimedia.org/T126406#2022071 (10hashar) [12:07:55] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2022072 (10hashar) [12:09:04] 10Continuous-Integration-Config: Timeouts in mediawiki-extensions-php53 - https://phabricator.wikimedia.org/T126406#2013605 (10hashar) This was partly due to Scribunto being added to it (as part of T125050) which caused the test runtime to skyrocket (T126274). [12:09:33] 10Continuous-Integration-Config: Have CI set `$wgScribuntoDefaultEngine = 'luasandbox` to speed up parser tests - https://phabricator.wikimedia.org/T126670#2022084 (10hashar) [12:09:35] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 5Patch-For-Review: Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050#1973302 (10hashar) [12:10:12] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 5Patch-For-Review: Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050#1973302 (10hashar) [12:13:42] 10Continuous-Integration-Infrastructure: Zuul seems to be running slower - https://phabricator.wikimedia.org/T118083#2022103 (10hashar) 5Open>3declined a:3hashar That is definitely https://github.com/openstack-infra/zuul/commit/5241b883711a1d1eb864fd746b84e634c75d26f1 which @Paladox found. The reason is G... [12:45:41] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022175 (10hashar) At least tex seems to be able to generate something: ``` --- [Math] Start rendering $\Sampi\sampi$ in mode png [Math]... [12:53:23] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 7Blocked-on-Operations, and 2 others: Beta: submodule update reverts new portals commits - https://phabricator.wikimedia.org/T126061#2022187 (10hashar) 5Open>3Resolved I am not sure why beta cluster is being used for live hack. The p... [12:55:41] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/269865 (owner: 10Paladox) [12:56:40] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/270125 (owner: 10Paladox) [12:57:17] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 7Blocked-on-Operations, and 2 others: Beta: submodule update reverts new portals commits - https://phabricator.wikimedia.org/T126061#2022201 (10hashar) [12:57:19] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 3Discovery-Portal-Sprint: Automatically deploy Wikimedia portals to Beta Cluster - https://phabricator.wikimedia.org/T124848#2022200 (10hashar) [12:58:33] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 3Discovery-Portal-Sprint: Automatically deploy Wikimedia portals to Beta Cluster - https://phabricator.wikimedia.org/T124848#1968218 (10hashar) Since the live hacks made on beta cluster got removed automatically and we now rebase ( T126... [13:15:18] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022238 (10Paladox) Maybe look at the differences between Precise and Trusty. [13:50:06] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022297 (10Physikerwelt) Too bad. Yet another reason to make progress on https://phabricator.wikimedia.org/T74240 [13:53:22] 10Continuous-Integration-Infrastructure, 10Math: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022298 (10Physikerwelt) e9dabb19e4c27bf23d3c2a3629474562 that should be the name of the tmp file in /mnt/home/jenkins-deploy/tmpfs/jenki... [14:48:58] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022343 (10hashar) >>! In T126422#2022298, @Physikerwelt wrote: > e9dabb19e4c27bf23d3c2a3629474562 that should be th... [14:53:12] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022352 (10hashar) ``` [MathTexvc] HASHAR file does not exist /mnt/home/jenkins-deploy/tmpfs/jenkins-1/8308ee5003aa3... [15:01:46] (03CR) 10JanZerebecki: [C: 04-1] "Some comments inline, otherwise looks good." (032 comments) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247056 (owner: 10XZise) [15:07:36] (03CR) 10JanZerebecki: [C: 04-1] "Can avsc validate its files far further than a simple json decoder?" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/265940 (owner: 10Paladox) [15:27:19] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022477 (10Physikerwelt) @hashar: You tested \Digamma which expects `5cfd6e5df6c87798542dca2e22c1e7cb`. `8308ee5003a... [15:27:31] 15:24:10 Warning: Task "karma:main" failed. Use --force to continue. [15:27:56] hashar: known problem ? [15:28:06] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/30648/console [15:30:16] thedj: that is qunit failing for some reason [15:30:47] 00:03:21.950 12 02 2016 15:24:09.799:WARN [Chromium 48.0.2564 (Ubuntu 0.0.0)]: Disconnected (1 times), because no message in 60000 ms. [15:30:52] a few lines above [15:31:01] while doing: 00:02:56.171 12 02 2016 15:23:44.021:DEBUG [proxy]: proxying request - /jenkins-mediawiki-extensions-qunit-30648/load.php?debug=false&lang=en&modules=jquery.effects.blind%2Ccore%7Cjquery.ui.autocomplete%2Ccore%2Cmenu%2Cposition%2Cwidget%7Cjquery.ui.core.styles&skin=fallback&version=278772a09135 to localhost:9412 [15:31:13] maybe it shows up the debug log under https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/30648/ [15:31:59] thedj: I would just "recheck" it [15:38:54] hashar: totally not urgent but i noticed the "mediawiki-core" link is broken on https://integration.wikimedia.org/cover/ [15:39:11] wondering if the link is just wrong or is correct and coverage is missing [15:39:14] ? [15:46:09] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022513 (10hashar) Eekkkk. So log output is proper: ``` [Math] Start rendering $\Digamma$ in mode png [Math] TeX: /... [15:46:21] aude: oh [15:46:28] aude: yeah the coverage no more generate [15:46:50] :( [15:46:52] sad [15:46:56] Precise? [15:47:00] php53 [15:47:09] it has been broken somehow a while back [15:47:19] and it is tied on Precise iirc [15:47:32] :( [15:47:51] ah https://phabricator.wikimedia.org/T125876 [15:47:57] MediaWiki test coverage kills Zend 5.3 engine (ex: zend_mm_heap corrupted), switch it to other instance [15:48:11] lets move it and see what happens [15:50:22] (03PS1) 10Hashar: mediawiki-core-code-coverage to Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) [15:51:50] 10Continuous-Integration-Config, 5Patch-For-Review: MediaWiki test coverage kills Zend 5.3 engine (ex: zend_mm_heap corrupted), switch it to other instance - https://phabricator.wikimedia.org/T125876#2022532 (10Krinkle) Beware that there were some issues with PHPUnit's coverage tool under HHVM. It'll need to b... [15:52:21] (03CR) 10Krinkle: [C: 031] "Yay" [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) (owner: 10Hashar) [15:52:38] Krinkle: I forgot PHP_BIN :-D [15:54:36] (03PS2) 10Hashar: mediawiki-core-code-coverage to Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) [15:54:59] hmm [15:55:04] somehow PHP_BIN is not needed ... [15:56:15] (03PS3) 10Hashar: mediawiki-core-code-coverage to Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) [15:56:43] (03CR) 10Hashar: "Back to PS1. PHP_BIN is not needed." [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) (owner: 10Hashar) [15:57:29] Krinkle: some recent PHPUnit versions can do coverage with hhvm and I think xhprof [15:57:33] never looked at it though [15:58:15] 10Continuous-Integration-Config, 5Patch-For-Review: MediaWiki test coverage kills Zend 5.3 engine (ex: zend_mm_heap corrupted), switch it to other instance - https://phabricator.wikimedia.org/T125876#2022565 (10hashar) Switched to Trusty, job is running on https://integration.wikimedia.org/ci/job/mediawiki-cor... [15:58:24] (03CR) 10Hashar: [C: 04-1] "Switched to Trusty, job is running on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/1827/" [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) (owner: 10Hashar) [16:00:28] hashar: I'd stick with zend on trusty for now [16:01:51] yeah [16:03:01] labs under maintenance. CI jobs might not trigger anymore due to lack of instances to run them in. [16:10:14] (03PS1) 10Zfilipin: Separated RelatedArticles Selenium jobs to desktop and mobile [integration/config] - 10https://gerrit.wikimedia.org/r/270309 (https://phabricator.wikimedia.org/T120715) [16:12:07] (03CR) 10Zfilipin: "The job will probably not work until this is merged:" [integration/config] - 10https://gerrit.wikimedia.org/r/270309 (https://phabricator.wikimedia.org/T120715) (owner: 10Zfilipin) [16:13:58] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022619 (10Physikerwelt) The only difference I noticed in the output that it's `/mnt/home/jenkins-deploy/tmpfs/jenki... [16:15:32] !log the pool of CI slaves is exhausted, no more jobs running (scheduled labs maintenance) [16:27:02] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022643 (10hashar) > The only difference I noticed in the output that it's /mnt/home/jenkins-deploy/tmpfs/jenkins-0... [16:31:40] !log bd808 added support for saltbot to update tasks automagically!!!! T108720 [16:31:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:31:55] https://phabricator.wikimedia.org/T108720#2022666 [16:31:58] neat [16:39:16] (03CR) 10Ori.livneh: [C: 032] Add Generic.Arrays.DisallowLongArraySyntax to ruleset, autofix this repo [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/269612 (owner: 10Legoktm) [16:46:06] hashar: I was wondering if it was both sal's (prod and us), neat! [16:51:38] !log running sudo salt '*' -b '10%' deploy.fixurl to fix deployment-prep trebuchet urls [16:51:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:56:03] (03Merged) 10jenkins-bot: Add Generic.Arrays.DisallowLongArraySyntax to ruleset, autofix this repo [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/269612 (owner: 10Legoktm) [17:03:28] bd808: next step when we move to Differential is commenting on Diffs when mentioned, which would be great for SWAT deploys. [17:03:41] wait, I guess we more care about the task there, too.... [17:03:55] either should be easy to add [17:04:30] yeah, either way, awesome fun project :) [17:04:33] the code will need refactoring at some point, but such is life [17:04:51] thanks for a ray of sunshine in the morning :) [17:05:15] :) I've been trying to do fun things instead of feeling pissed/grumpy about other things [17:06:14] next up is this feature requested by twentyafterfour -- https://github.com/bd808/tools-stashbot/issues/5 [17:09:02] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022820 (10Paladox) @hashar is there any way for us to disable that test temporarily I mean disable the test that fa... [17:10:16] !log Nodepool back at spawning instances. contintcloud has been migrated in wmflabs [17:10:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:10:33] yay (re contint) [17:11:04] hashar: Would i be able to try npm-node-4.2 on skins or extensions in mediawiki. [17:11:13] paladox: no [17:11:16] bd808: ah yeah, that'd be nice too [17:11:18] hashar: Ok. [17:11:27] paladox: it is too early [17:11:34] will get the mediawiki/services migrated there [17:11:50] then bump the # of instances and mass/bulk push the 'npm' job [17:12:07] hashar: Oh ok. Should i change mediawiki/services to npm-node-4.2. [17:13:04] hashar: It looks like https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/1827/console has stalled. [17:13:40] paladox: no please let me handle the npm node-4.2 stuff [17:13:47] it is scary enough and undocumented (my fault for that) [17:13:56] hashar: Ok. [17:14:15] hashar: Why not try it in experimental: pipeline so the risks are minimal. [17:14:21] gotta doc it for the rest of #releng anywy [17:14:39] just pretend it does not exist ;-} [17:15:14] hashar: Ok. [17:15:42] (03CR) 10Paladox: "@JanZerebecki I'm not sure." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/265940 (owner: 10Paladox) [17:16:44] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022920 (10Physikerwelt) @Hashar, can you try to run the latex command that is actually run by the texvc? Create a... [17:19:10] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022933 (10Physikerwelt) @Paladox the more tests we have the better;-) But all the files latex etc will be gone with... [17:19:21] !log get rid of integration-dev it is broken somehow [17:19:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:19:54] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022936 (10Paladox) @Physikerwelt ok. [17:20:20] (03PS2) 10Paladox: Add file extension avsc to json-lint.php file [integration/jenkins] - 10https://gerrit.wikimedia.org/r/265940 [17:21:15] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 0.12 or Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2022946 (10Krinkle) 3NEW [17:22:10] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [17:22:18] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022954 (10Physikerwelt) @hashar for testing it's actually better to leave out the \nonstopmode [17:24:06] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022966 (10hashar) ``` hashar@integration-slave-trusty-1001:~$ latex T126422.tex This is pdfTeX, Version 3.1415926-... [17:24:39] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 0.12 or Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2022971 (10Paladox) [17:26:26] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#2022988 (10matmarex) [17:26:29] 7Browser-Tests, 6Multimedia, 10UploadWizard: Fix failed UploadWizard browsertests Jenkins job - https://phabricator.wikimedia.org/T94161#2022984 (10matmarex) 5Open>3declined a:3matmarex Mark says that nobody cares and that we're going to use something new for browser tests soon. Pretty sure the old one... [17:27:13] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2022991 (10hashar) That points to OCG commit https://gerrit.wikimedia.org/r/#/c/107585/ ``` Add missing texlive-gen... [17:29:47] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2023001 (10JanZerebecki) That file is in `texlive-generic-extra`, so maybe the solution is to fix the puppet code to... [17:30:43] zeljkof, marxarelli|afk, how is that new JS-based browser test thing looking? Any way I can help? [17:32:35] !log adding texlive-generic-extra on CI slaves by cherry picking https://gerrit.wikimedia.org/r/#/c/270322/ - T126422 [17:32:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:32:38] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2023008 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVLWiqEi-0X0Il_jx... [17:35:36] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2023026 (10hashar) ``` $ latex T126422.tex This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013/Debian) re... [17:35:42] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [17:36:29] !log salt -v '*slave-trusty*' cmd.run 'apt-get -y install texlive-generic-extra' # T126422 [17:36:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:36:34] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2023027 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVLWjjLfhQaf1CQcC... [17:38:18] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2023042 (10hashar) a:3hashar Should be good now ? [17:43:55] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [17:49:51] (03CR) 10Paladox: [C: 031] "Works now." [integration/config] - 10https://gerrit.wikimedia.org/r/270305 (https://phabricator.wikimedia.org/T125876) (owner: 10Hashar) [18:09:47] 10Beta-Cluster-Infrastructure: rebuild deployment-bastion on trusty - https://phabricator.wikimedia.org/T126537#2023268 (10dduvall) [18:10:52] 3Scap3, 10scap, 6Phabricator, 7WorkType-NewFunctionality: Deploy Phabricator with scap3 - https://phabricator.wikimedia.org/T114363#2023273 (10mmodell) [18:10:56] 3Scap3: include refreshCdbJsonFiles in scap's debian package - https://phabricator.wikimedia.org/T126660#2023271 (10mmodell) 5Open>3Resolved [18:14:43] 3Scap3: include refreshCdbJsonFiles in scap's debian package - https://phabricator.wikimedia.org/T126660#2023297 (10mmodell) 5Resolved>3Open [18:14:45] 3Scap3, 10scap, 6Phabricator, 7WorkType-NewFunctionality: Deploy Phabricator with scap3 - https://phabricator.wikimedia.org/T114363#2023298 (10mmodell) [18:30:04] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 2.36 ms [18:39:04] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 7Blocked-on-Operations, and 2 others: Beta: submodule update reverts new portals commits - https://phabricator.wikimedia.org/T126061#2023357 (10ksmith) @hashar: I'm a bit out of the technical loop, but my understanding is that the portal... [18:44:09] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [18:46:06] 10Continuous-Integration-Infrastructure: puppet-compiler does not create change.*.pson file - https://phabricator.wikimedia.org/T126796#2023382 (10scfc) 3NEW [18:48:27] twentyafterfour: Hey. Would be cool if you could create a followup task for T120013 for the (next+1) Phab upgrade so I could mark some items as Blocked By already. (And I don't want to interfere with your workstyle hence asking you here.) TIA [18:55:47] andre__: sure, you can always feel free to create one if I haven't done so [18:55:55] but I'll do that now [18:55:56] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 2.36 ms [18:57:19] 3Scap3, 10scap, 7Epic: EPIC: Future Deployment Tooling - https://phabricator.wikimedia.org/T101023#2023463 (10dduvall) 5Open>3Resolved a:3dduvall The initial discussion/planning for this was completed long ago and the services deploy MVP is already tracked in {T109535}. Let's close this out. [19:01:54] These full test times are insane still. Waiting 21 minutes now for a backport merge that looks like it's going to eventually fail and need to done again [19:03:17] What started making Math take so long? It seems to be what was plugging up the gate-and-submit queue [19:04:23] RECOVERY - Puppet failure on integration-slave-trusty-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:32] Math depends upon Wikibase now [19:08:01] 10Deployment-Systems, 6Release-Engineering-Team: Implement "new weekly release deploy duration" KPI - https://phabricator.wikimedia.org/T108742#2023581 (10dduvall) [19:08:03] 6Release-Engineering-Team, 3Scap3, 10scap, 7WorkType-NewFunctionality: Instrument scap for "scap duration" KPI - https://phabricator.wikimedia.org/T108743#2023579 (10dduvall) 5Open>3declined I don't see this on our list of KPIs for upcoming quarters so closing this out for now. [19:09:59] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [19:10:23] Project beta-scap-eqiad build #89663: 04FAILURE in 5 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89663/ [19:15:05] 3Scap3: include refreshCdbJsonFiles in scap's debian package - https://phabricator.wikimedia.org/T126660#2023617 (10mmodell) @fgiunchedi: Can you rebuild the scap package? [19:15:31] 3Scap3, 7Blocked-on-Operations: include refreshCdbJsonFiles in scap's debian package - https://phabricator.wikimedia.org/T126660#2023621 (10mmodell) [19:15:53] 3Scap3, 7Blocked-on-Operations: include refreshCdbJsonFiles in scap's debian package - https://phabricator.wikimedia.org/T126660#2023624 (10mmodell) a:3fgiunchedi [19:16:29] 3Scap3, 7Blocked-on-Operations: rebuild scap debian package (we forgot to include refreshCdbJsonFiles) - https://phabricator.wikimedia.org/T126660#2023626 (10mmodell) [19:21:43] 3Scap3, 10scap, 7WorkType-NewFunctionality: Scap3 check to monitor logstash and detect changes in error frequency - https://phabricator.wikimedia.org/T110068#2023652 (10mobrovac) While this has been moved into the MediaWiki MVP column, I'd like to voice that services would benefit from this too and has in fa... [19:23:52] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.30 ms [19:24:09] 10Continuous-Integration-Infrastructure: puppet-compiler does not create change.*.pson file - https://phabricator.wikimedia.org/T126796#2023661 (10Dzahn) There is no "change catalog" because the puppet runs fail, therefore not creating any change catalog. [19:24:30] 3Scap3, 7Blocked-on-Operations: rebuild scap debian package (we forgot to include refreshCdbJsonFiles) - https://phabricator.wikimedia.org/T126660#2023662 (10mmodell) a:5fgiunchedi>3None [19:26:26] 3Scap3, 10scap: Give tasks clearer names - https://phabricator.wikimedia.org/T126372#2023677 (10dduvall) p:5Triage>3Normal [19:27:00] Dear releng people, CI is still broken for extensions :( https://phabricator.wikimedia.org/T126699 [19:27:29] 3Scap3, 7Blocked-on-Operations: rebuild scap debian package (we forgot to include refreshCdbJsonFiles) - https://phabricator.wikimedia.org/T126660#2023690 (10mmodell) The package got tested fairly extensively on beta: we replaced deployment-bastion with a fresh deployment instance, deployment-tin, and went th... [19:30:45] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [19:31:37] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#2023699 (10mmodell) [19:31:52] 3Scap3, 10scap, 7Epic: EPIC: Scap3 should implement the services team requirements - https://phabricator.wikimedia.org/T109535#2023705 (10dduvall) 5Open>3Resolved a:3dduvall All requirements have been implemented. We're now working on adoption. [19:33:11] 3Scap3, 10scap: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#2023710 (10dduvall) p:5Triage>3High [19:33:20] 3Scap3, 10scap: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#1636723 (10dduvall) p:5High>3Normal [19:35:21] 3Scap3, 10scap, 7Documentation: Document Scap3's `--limit` flag - https://phabricator.wikimedia.org/T118745#2023733 (10dduvall) p:5Triage>3Normal [19:39:01] 6Release-Engineering-Team, 3Scap3, 10scap, 7Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#2023774 (10dduvall) p:5Triage>3Normal [19:39:03] 6Release-Engineering-Team, 3Scap3, 10scap, 7Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#1801285 (10dduvall) [19:39:46] 3Scap3, 10scap: File ownership differences between Scap3 and Trebuchet - https://phabricator.wikimedia.org/T116632#2023779 (10dduvall) p:5Triage>3Normal [19:39:50] 3Scap3, 10scap, 7Documentation: End user tutorial docs for Scap - https://phabricator.wikimedia.org/T118738#2023789 (10dduvall) p:5Triage>3Normal [19:39:57] 3Scap3, 10scap: File ownership differences between Scap3 and Trebuchet - https://phabricator.wikimedia.org/T116632#1754338 (10dduvall) p:5Triage>3Normal [19:40:25] 3Scap3, 10scap: Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#2023796 (10dduvall) p:5Triage>3Low [19:41:22] 10Deployment-Systems, 10scap: sync-masters slow on mira - https://phabricator.wikimedia.org/T125108#2023802 (10dduvall) [19:42:41] 3Scap3, 10scap: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#2023805 (10dduvall) p:5Triage>3Normal [19:44:40] jzerebecki: Can i add you too the patches that migrate to npm and composer please. Such as patches that remove jslint and replaced with either jshint and jsonlint. [19:46:39] Yippee, build fixed! [19:46:39] Project beta-scap-eqiad build #89666: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89666/ [19:47:39] 3Scap3, 10scap: scap3 host restart batching should allow for delay between batches - https://phabricator.wikimedia.org/T122914#2023830 (10dduvall) This might be tricky since scap3 doesn't do batches per se but concurrency levels. A post-stage check could be leveraged to induce a delay, however. ```lang=yaml c... [19:48:44] 3Scap3, 10scap: scap3 host restart batching should allow for delay between batches - https://phabricator.wikimedia.org/T122914#2023832 (10dduvall) p:5Triage>3Normal [19:48:54] 3Scap3, 10scap: scap3 host restart batching should allow for delay between batches - https://phabricator.wikimedia.org/T122914#1916651 (10dduvall) p:5Normal>3Low [19:49:33] 3Scap3: Proof-of-concept: sync l10n cache with git-annex + zsync - https://phabricator.wikimedia.org/T126805#2023837 (10mmodell) 3NEW a:3mmodell [19:49:50] 3Scap3: Proof-of-concept: sync l10n cache with git-annex + zsync - https://phabricator.wikimedia.org/T126805#2023847 (10mmodell) p:5Triage>3Low [19:51:08] 10Deployment-Systems, 10scap: sync-wikiversions not syncing wikiversions.json with mira - https://phabricator.wikimedia.org/T121585#2023852 (10dduvall) [19:51:20] 3Scap3, 10scap: Bring co-master / fanout capabilities to `deploy` and friends - https://phabricator.wikimedia.org/T121276#2023854 (10dduvall) p:5Triage>3High [19:51:41] 3Scap3, 10scap, 10RESTBase-Cassandra: Deploy Cassandra with scap3 - https://phabricator.wikimedia.org/T116340#2023861 (10dduvall) p:5Normal>3High [19:51:47] 3Scap3, 10scap, 10Mathoid: Deploy Mathoid with scap3 - https://phabricator.wikimedia.org/T116338#2023864 (10dduvall) p:5Normal>3High [19:51:55] 3Scap3, 10scap, 10Citoid: Deploy Citoid with scap3 - https://phabricator.wikimedia.org/T116337#2023867 (10dduvall) p:5Normal>3High [19:52:45] 3Scap3, 10scap, 10ContentTranslation-cxserver: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2023873 (10dduvall) p:5Normal>3High [19:54:02] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms [19:55:05] 3Scap3, 10scap, 5Patch-For-Review: Support smooth transitions from Trebuchet managed deploys - https://phabricator.wikimedia.org/T113107#2023892 (10dduvall) 5Open>3Resolved a:3dduvall [19:57:13] 3Scap3, 10scap, 10ContentTranslation-cxserver: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2023904 (10dduvall) p:5High>3Normal [19:57:24] 3Scap3, 10scap, 10Citoid: Deploy Citoid with scap3 - https://phabricator.wikimedia.org/T116337#2023906 (10dduvall) p:5High>3Normal [19:57:31] 3Scap3, 10scap, 10Mathoid: Deploy Mathoid with scap3 - https://phabricator.wikimedia.org/T116338#2023910 (10dduvall) p:5High>3Normal [19:57:42] 3Scap3, 10scap, 10RESTBase-Cassandra: Deploy Cassandra with scap3 - https://phabricator.wikimedia.org/T116340#2023912 (10dduvall) p:5High>3Normal [20:01:44] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 0.12 or Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2023931 (10Jdforrester-WMF) [20:02:08] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [20:08:09] 3Scap3, 10scap: Allow scap3 to read target host list from stdin - https://phabricator.wikimedia.org/T122913#2023939 (10mmodell) a:3mmodell I can work on this [20:14:42] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 0.12 or Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2023945 (10GWicke) FWIW, nodes services are basically all on node 4 by now. I don't think there is much value in bothering with 0.12. [20:18:43] MarkTraceur: sorry, weekend starts early in Europe :) [20:19:00] zeljkof: I sympathize :) [20:20:11] This is what we have so far [20:20:14] https://gerrit.wikimedia.org/r/#/c/256404/ [20:20:36] Nifty [20:41:31] 10Continuous-Integration-Infrastructure: Passing jenkins job reported as failing - https://phabricator.wikimedia.org/T126810#2024015 (10Tgr) 3NEW [20:44:30] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2024026 (10hashar) 5Open>3Resolved I have 'recheck' a few changes and the tests pass. The root cause is that Ubu... [20:44:40] 10Continuous-Integration-Infrastructure: Passing jenkins job reported as failing - https://phabricator.wikimedia.org/T126810#2024028 (10Tgr) I missed the error at the very end: ``` 20:28:11 + /srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh 20:28:11 ERROR 1269 (HY000) at line 1: Can't revoke al... [20:52:39] (03Abandoned) 10Reedy: Revert "Remove broken special_extensions behavior" [tools/release] - 10https://gerrit.wikimedia.org/r/265136 (owner: 10Reedy) [20:57:11] 10Continuous-Integration-Infrastructure, 10Math, 5Patch-For-Review: Texlive on CI Trusty slaves lacks Ancient Greek causing Math test fail for php55 - https://phabricator.wikimedia.org/T126422#2024038 (10Jdforrester-WMF) Thanks Antoine! [21:05:31] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2024057 (10Jdforrester-WMF) [21:05:33] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 0.12 or Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2024056 (10Jdforrester-WMF) [21:05:53] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#1819116 (10Jdforrester-WMF) Is there a reason this isn't in infrastructure? [21:06:57] 10Continuous-Integration-Infrastructure, 7Upstream: Npm crash: "code EBADF; errno 9; EBADF, fstat" - https://phabricator.wikimedia.org/T93425#2024060 (10Jdforrester-WMF) [21:20:16] Project beta-update-databases-eqiad build #6432: 04FAILURE in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6432/ [21:42:39] hashar: Could we bring back these type test mwext-WikibaseQualityExternalValidation back to WikibaseQualityExternalValidation but blacklist it so it dosent run on master branch and black list the unittests composer test to not run on branches REL1_26 or lower. [21:42:48] It is currently failing on https://gerrit.wikimedia.org/r/#/c/268807/ [21:42:57] jzerebecki: ^^ [21:44:44] hashar: Npm fails for me https://integration.wikimedia.org/ci/job/mediawiki-core-npm/9392/console [21:45:13] paladox: The package grunt@1.0.0-rc1 does not satisfy its siblings' peerDependencies requirements! [21:45:16] that package is broken [21:45:33] hashar: Oh. [21:45:49] just stop blindly upgrading packages [21:46:06] there is no need to upgrade unless you need a bug to be fixed or a new feature [21:46:18] and rc == release candidate, hence it is hardly stable [21:46:29] and even 1.0.0 is probably to have a bunch of bugs [21:46:42] hashar: Ok. [21:49:40] 3Scap3: deploy-local (TargetContext) should not default to utils.get_real_username() - https://phabricator.wikimedia.org/T126489#2024185 (10dduvall) [21:49:49] (03PS2) 10Paladox: [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [21:50:31] 3Scap3: deploy-local (TargetContext) should not default to utils.get_real_username() - https://phabricator.wikimedia.org/T126489#2024187 (10dduvall) [21:51:41] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:05:52] greg-g: I am still banned from the other chan :( [22:15:30] (03PS3) 10Paladox: [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:16:26] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [22:17:03] cache-rsync I think I deleted it [22:17:52] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:18:37] (03PS4) 10Paladox: [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:19:02] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [22:21:15] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:22:03] (03PS5) 10Paladox: [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:24:07] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [22:24:25] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:24:33] I'm seeing random CI failures due to randomly missing files/directories... [22:25:14] 22:23:33 Query: SHOW TABLES [22:25:14] 22:23:33 Function: MediaWikiTestCase::listTables [22:25:14] 22:23:33 Error: 1049 Unknown database 'jenkins_u2_mw' (127.0.0.1:3306) [22:25:18] mysql is also going away? [22:25:20] (03PS6) 10Paladox: [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:26:47] hashar: ^ [22:26:54] greg-g: mini phabricator maintenance on monday with ~ 5-10 minutes downtime. Where should I announce such a thing? wikitech-l? [22:27:19] yeah [22:27:58] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add a sqlite variant of extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/269653 (owner: 10JanZerebecki) [22:28:57] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.50 ms [22:30:03] 10Continuous-Integration-Infrastructure, 7Upstream: Npm crash: "code EBADF; errno 9; EBADF, fstat" - https://phabricator.wikimedia.org/T93425#2024305 (10Krinkle) 5stalled>3declined a:3Krinkle Haven't seen it since. Closed upstream as well. Lots of node and npm releases since then. [22:32:26] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2024340 (10hashar) [22:33:08] (03CR) 10Paladox: "Please revert this and instead use the skip if to blacklist the master branch for these old tests so they only one on REL1_26 or below." [integration/config] - 10https://gerrit.wikimedia.org/r/269652 (https://phabricator.wikimedia.org/T126441) (owner: 10JanZerebecki) [22:33:14] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#1819116 (10hashar) >>! In T119143#2024057, @Jdforrester-WMF wrote: > Is there a reason this isn't i... [22:35:51] PROBLEM - Host integration-dev is DOWN: PING CRITICAL - Packet loss = 100% [22:40:01] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2024365 (10hashar) The exact message is: ``` $ apt-cache policy openjdk-7-jre-headless W: Duplicate sources.list entry http://mirrors.wikimed... [22:40:35] does a recheck work on +2 these days ? [22:40:40] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2024374 (10Krinkle) >>! In T119143#1843027, @hashar wrote: > A big stopper is that some repositorie... [22:42:43] thedj: no, just remove your vote and re +2 it [22:44:33] oh duh, i had +2 the previous PS [22:45:06] thcipriani: if you are still around. Do you remember the SoS task that is blocked on us related to node 4.2 ? [22:45:11] I am writing a status update [22:45:30] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: rebuild deployment-bastion on trusty - https://phabricator.wikimedia.org/T126537#2024386 (10thcipriani) = What we did = == Updates to puppet == * Replace all instances where deployment-bastion was hard-coded with deployment-tin ** same for IP (10.68.16.58 → 1... [22:46:30] hashar: yep https://phabricator.wikimedia.org/T125003 [22:46:59] is the task we were asked about specifically at SoS [22:49:07] been looking at https://phabricator.wikimedia.org/T119143 [22:49:30] thcipriani: ah mine is for CI [22:49:35] the other is for beta cluster .. [22:50:01] I am not going to rant how folks migrate production first without notice [22:50:08] :D [22:50:09] then escaladate about ci / beta etc not following [22:50:27] RIGHT WHEN WE HAVE TWO WEEKS OF MADNESS [22:50:45] (i am not upset, despite caps) [22:52:00] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2024408 (10hashar) So the lame status update is: **Feb 1 - Feb 5** That was the week I had to rid... [22:52:10] https://phabricator.wikimedia.org/T119143#2024408 status update [22:53:16] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2008504 (10hashar) [22:53:18] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Provision pkg-config on Nodepool instances - https://phabricator.wikimedia.org/T126230#2008087 (10hashar) [22:53:52] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.20 ms [22:54:39] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Provision pkg-config on Nodepool instances - https://phabricator.wikimedia.org/T126230#2024418 (10hashar) That is the same provisioning issue as T126246 namely: ``` jenkins@ci-jessie-wikimedia-3... [22:57:09] (03PS1) 10Hashar: dib: add non-free, contribs components [integration/config] - 10https://gerrit.wikimedia.org/r/270433 (https://phabricator.wikimedia.org/T126246) [23:00:38] 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Provision pkg-config on Nodepool instances - https://phabricator.wikimedia.org/T126230#2024431 (10hashar) [23:00:45] 5Continuous-Integration-Scaling, 5Patch-For-Review: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2024433 (10hashar) [23:01:25] 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Provision pkg-config on Nodepool instances - https://phabricator.wikimedia.org/T126230#2024436 (10hashar) a:3hashar [23:09:18] 10Deployment-Systems, 6Performance-Team, 10Traffic, 6operations, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2024490 (10Krinkle) [23:13:27] 10Beta-Cluster-Infrastructure: Beta cluster bits should not cache static-master for three weeks - https://phabricator.wikimedia.org/T90983#2024523 (10Krinkle) [23:13:30] 10Deployment-Systems, 6Performance-Team, 10Traffic, 6operations, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2024524 (10Krinkle) [23:26:57] 10Beta-Cluster-Infrastructure, 10Kartographer, 3Discovery-Maps-Sprint: Deploy Kartographer to beta cluster - https://phabricator.wikimedia.org/T126829#2024553 (10MaxSem) 3NEW a:3MaxSem [23:27:10] yurik, ^^^^^^^^^^^ [23:27:13] :P [23:27:38] MaxSem, \o/ [23:27:44] want to do it? [23:27:52] not right now :P [23:28:00] why not? :) [23:28:06] we can totally launch it already [23:28:16] Friiidayyyy dude [23:28:23] and PM at that ;) [23:30:02] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [23:31:21] 5Continuous-Integration-Scaling, 5Patch-For-Review: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2024571 (10hashar) Better: ``` $ virt-cat -a image-jessie-20160212T225848Z.qcow2 /etc/apt/sources.list deb http://mirrors.wikimedia.org/debian/ jessie main non-fre... [23:32:01] (03PS2) 10Hashar: dib: add non-free, contribs components [integration/config] - 10https://gerrit.wikimedia.org/r/270433 (https://phabricator.wikimedia.org/T126246) [23:32:31] (03CR) 10Hashar: "Nit: changed order of components." [integration/config] - 10https://gerrit.wikimedia.org/r/270433 (https://phabricator.wikimedia.org/T126246) (owner: 10Hashar) [23:33:15] 5Continuous-Integration-Scaling, 5Patch-For-Review: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2024573 (10mobrovac) Note that `deb http://mirrors.wikimedia.org/debian/ jessie-backports main contrib non-free` appears twice, once in `/etc/apt/sources.list` and... [23:35:22] MaxSem, its ok for beta depl ;) [23:35:42] if greg-g permits... [23:36:00] MaxSem, do we need greg-g permition for labs depl?? [23:39:02] he is probably off [23:39:06] you can do it [23:39:17] would need some changes in mediawiki-config [23:39:28] and please please make sure the jenkins jobs are all fine [23:39:38] hashar, jenkins jobs? [23:39:41] https://integration.wikimedia.org/ci/view/Beta/ [23:40:02] we have jobs that git pull mediawiki/extensions / mediawiki/core & vendor every x minutes [23:40:18] another one doing database upgrade every hour (apparently it broke 19 minutes ago for some reason) [23:40:32] hashar, want to help with kartographer depl? :) [23:40:37] beta-scap-eqiad is triggerd when code has been pulled. It run scap / rebuild l10n etc [23:40:38] its simple - no DB changes [23:40:47] if it is broken, beta cluster code is no more deployed [23:41:12] yeah just do it. but make sure jobs run fine [23:41:47] once the change merges in mediawiki-config that triggers https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ [23:41:49] which pull [23:41:57] and that job then trigger https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [23:42:05] so both should be green once you hve finished [23:42:23] https://logstash-beta.wmflabs.org/ can help as well [23:42:29] MaxSem: yurik all of that^^ :) [23:42:44] hashar, hold on, first we need to alter mw config [23:43:27] i'm a simple person with no memory beyond one line back [23:43:33] small buffer [23:44:25] twentyafterfour: yeah, wikitech-l/engineering@ [23:45:11] (sorry, was at an appt) [23:45:21] TIL about virt-rescue [23:47:14] hashar, https://gerrit.wikimedia.org/r/#/c/270441/ [23:47:17] like that? [23:48:42] yurik, also need to wfLoadExtension() [23:50:15] MaxSem, thx, i'm silly, fixing settings [23:50:43] 5Continuous-Integration-Scaling, 5Patch-For-Review: Provision openjdk-7-jre-headless on Nodepool slaves - https://phabricator.wikimedia.org/T126246#2024593 (10hashar) Hooked in the image I build via: ``` $ virt-rescue -a image-jessie-20160212T225848Z.qcow2 $ mount /dev/sda1 /sysroot/ $ chroot /s... [23:50:59] had fun on friday evening [23:51:03] learned virt-rescue [23:51:05] wrote a report [23:51:18] time to sleep [23:51:32] yurik: it is 1am here in CET. so that is week end time [23:51:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-ResourceLoader, 6Performance-Team, 5Patch-For-Review: Beta cluster bits should not cache static-master for three weeks - https://phabricator.wikimedia.org/T90983#2024594 (10Krinkle) a:3Krinkle [23:51:47] you might want to wait on monday and get the stuff reviewed [23:51:57] hashar, even for labs? [23:52:01] yeah [23:52:10] i thought labs can be played on weekend as well [23:52:18] if i notice it is broken because of kartographer I will just revert it really [23:52:21] in case it break [23:52:25] but maybe it will not :) [23:52:31] sec, almost done :) [23:52:37] too late [23:52:45] been a 50+ hours week already :D [23:53:12] maybe some other swat / mediawiki-config can assist [23:54:05] !log beta cluster broken since 20:30 UTC https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor havent looked [23:54:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:54:22] Fatal error: Class undefined: GWToolset\Config in /srv/mediawiki/wmf-config/CommonSettings.php on line 1902 [23:54:32] Class undefined: GWToolset\Config in /srv/mediawiki/wmf-config/CommonSettings.php on line 1902 [23:54:33] bbbeeee [23:54:33] MaxSem, hashar https://gerrit.wikimedia.org/r/#/c/270441/ [23:55:07] hashar, will it auto-deploy to beta cluster if we +2 it? [23:55:27] bee, someone broke it:( [23:55:34] (not kartographe) [23:56:39] 10Beta-Cluster-Infrastructure, 10MediaWiki-ResourceLoader, 6Performance-Team, 5Patch-For-Review: Beta cluster bits should not cache static-master for three weeks - https://phabricator.wikimedia.org/T90983#2024608 (10Krinkle) After T99096, caching these urls for 3 weeks is fine. In fact, it'll be 30 days. B...