[01:39:43] Yippee, build fixed! [01:39:43] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #482: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/482/ [02:08:30] 6Release-Engineering-Team, 3releng-201516-q3, 10Wikimedia-Developer-Summit-2016, 15User-greg: Code-review migration to Differential status/discussion - https://phabricator.wikimedia.org/T114320#1904418 (10JanZerebecki) > In fact, it's actually fairly difficult to lock down Git in the batshit insane ways Ge... [03:28:45] 10Continuous-Integration-Config, 5Patch-For-Review: Mark UserDailyContribs as archived in CI - https://phabricator.wikimedia.org/T122279#1904429 (10Liuxinyu970226) [05:00:18] 10Continuous-Integration-Infrastructure: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904447 (10Unicornisaurous) 3NEW [05:06:41] 10Continuous-Integration-Infrastructure: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904462 (10Unicornisaurous) [05:08:38] 7Browser-Tests, 10Continuous-Integration-Infrastructure, 10Browser-Tests-Infrastructure, 7Jenkins: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904447 (10Unicornisaurous) [05:09:47] 7Browser-Tests, 10Continuous-Integration-Infrastructure, 10Browser-Tests-Infrastructure, 7Jenkins: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904465 (10Unicornisaurous) [05:17:49] 7Browser-Tests, 10Continuous-Integration-Infrastructure, 10Browser-Tests-Infrastructure, 7Jenkins: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904466 (10Unicornisaurous) [05:21:36] 7Browser-Tests, 10Continuous-Integration-Infrastructure, 10Browser-Tests-Infrastructure, 7Jenkins: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904467 (10Unicornisaurous) [06:33:20] 10Continuous-Integration-Infrastructure: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904477 (10Legoktm) [10:00:52] !log restarted CI puppetmaster [10:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:05:58] 10Continuous-Integration-Infrastructure: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904499 (10hashar) I am merely acknowledging this issue. Unlikely to get it fixed anytime soon since I am traveling right now.. On a first diagno... [10:07:35] !log restarted Xvfb on trusty-1011 and rebooted trusty-1015. mediawiki-extensions-qunit randomly fails on some hosts ( https://phabricator.wikimedia.org/T122449 ) :( [10:07:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:07:48] !log no clue what is going on and I am traveling. Will look later tonight [10:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:35:59] RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0] [12:55:14] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #701: 04FAILURE in 1 min 13 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/701/ [12:59:28] Yippee, build fixed! [12:59:29] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #890: 09FIXED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/890/ [14:28:04] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #735: 04FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/735/ [18:46:37] 14:48:18 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [18:46:40] 14:48:18 Build step 'Execute shell' marked build as failure [18:46:56] on integration-slave-trusty-1015 [18:47:41] with https://gerrit.wikimedia.org/r/#/c/186757/13 [19:06:30] (03CR) 10Paladox: [C: 031] "This can be merged now." [integration/config] - 10https://gerrit.wikimedia.org/r/239681 (owner: 10Paladox) [19:07:58] (03Abandoned) 10Paladox: [WikibaseJavaScriptApi] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243597 (https://phabricator.wikimedia.org/T100654) (owner: 10Paladox) [19:37:22] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904882 (10Unicornisaurous) Looks like jenkins was playing tricks on me, as an //even-numbered// build machine has also [fail... [20:07:41] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904917 (10Paladox) Increasing the timeout may solve the problem until the actual problem is fixed. Some tests may fail but t... [20:15:41] merry christmas releng. just me, or is CI broken? [20:21:31] hmm, nevermind, was apparently transient. still looks worrying though. https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/24610/console from https://gerrit.wikimedia.org/r/#/c/260892/ [20:22:54] !sal [20:22:54] https://tools.wmflabs.org/sal/releng [20:27:16] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1015 : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock - https://phabricator.wikimedia.org/T122459#1904921 (10hashar) 3NEW [20:30:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [20:34:40] !log integration: cherry picked puppet patches https://gerrit.wikimedia.org/r/#/c/208024/ (raita role) and https://gerrit.wikimedia.org/r/#/c/204528/ (mysql on tmpfs) [20:34:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:38:20] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1015 : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock - https://phabricator.wikimedia.org/T122459#1904934 (10hashar) 5Open>3Resolved a:3hashar The patch `contint: Put mysql db on tmpfs for role::ci::slave::la... [20:38:22] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904937 (10hashar) [20:50:11] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904946 (10hashar) From one of the MediaWiki debug log running under Apache (which QUnit hits) such as https://integration.w... [20:58:43] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904953 (10Paladox) Thanks for tracing the problem or at least tracing where the problem is coming from could that mean that... [21:03:27] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904961 (10hashar) So it seems the proxy host `webproxy.eqiad.wmnet:8080` is unresponsive. Icinga has a Squid check: `TCP OK... [21:06:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-General-or-Unknown: Update grunt-jsonlint to 1.0.7 on all repos, on git Wikimedia or todo with Wikimedia - https://phabricator.wikimedia.org/T122460#1904966 (10Paladox) 3NEW [21:17:30] 10Continuous-Integration-Infrastructure, 6operations: Webproxy on carbon unreachable from labs instances since Dec 24 roughly 1am - https://phabricator.wikimedia.org/T122461#1904975 (10hashar) 3NEW [21:18:50] (03PS1) 10Paladox: [timeline] Add experimental tests to jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/261093 [21:20:11] hashar: Could you review https://gerrit.wikimedia.org/r/#/c/261093/ please so i could test https://gerrit.wikimedia.org/r/#/c/243936/ patch. [21:21:57] paladox: I would not waste anytime on timeline. It is a very old extension which is barely maintained anymore [21:22:18] hashar: Oh ok. [21:22:19] paladox: though it is still on production. So renaming the entry point would need some sync work with the Wikimedia cluster config [21:22:33] paladox: and I don't think anyone is going to spend time on it :D [21:22:46] paladox: so probably better to just ignore that specific extension :-} [21:22:52] (03PS1) 10Paladox: Add experimental tests to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/261094 [21:23:36] hashar: Ok. Could you review https://gerrit.wikimedia.org/r/261094 instead since it would help with other extensions that are maybe maintained but doint have any testextension tests. [21:23:41] please/ [21:23:45] paladox: and that last patch is wrong as well. The Zuul template 'npm' is solely for npm so it makes no sense to add unrelated jobs to the template [21:24:43] paladox: for extensions lacking the phptest, maybe you can bulk add mwext-testextension-hhvm to their experimental pipelines [21:24:46] experimental: [21:24:48] - mwext-testextension-hhvm [21:25:08] hashar: How would i bulk add them. [21:25:34] paladox: look for any projects defined in Zuul that lacks one of the phpunit tests templates ? [21:25:44] no idea how many are missing it though [21:26:02] hashar: Oh ok. Thanks will do. [21:29:15] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1904992 (10faidon) [21:29:17] 10Continuous-Integration-Infrastructure, 6operations: Webproxy on carbon unreachable from labs instances since Dec 24 roughly 1am - https://phabricator.wikimedia.org/T122461#1904987 (10faidon) 5Open>3Invalid a:3faidon See T122368. Why do you need to use the webproxy? Labs instances have Internet connect... [21:37:10] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 3releng-201516-q3, and 5 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#1904993 (10JanZerebecki) >>! In T119908#1904174, @Yurik wrote: > If we use GitHub, more develope... [21:37:14] (03PS1) 10Paladox: Update jshint to 2.8 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/261095 [21:40:27] (03PS1) 10Hashar: mwconf: no more set $wgHTTPProxy [integration/jenkins] - 10https://gerrit.wikimedia.org/r/261096 (https://phabricator.wikimedia.org/T122449) [21:41:10] !log integration: getting rid of $wgHTTPProxy https://gerrit.wikimedia.org/r/261096 (no more needed) [21:41:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:41:41] (03CR) 10Hashar: [C: 032] mwconf: no more set $wgHTTPProxy [integration/jenkins] - 10https://gerrit.wikimedia.org/r/261096 (https://phabricator.wikimedia.org/T122449) (owner: 10Hashar) [21:42:44] (03Merged) 10jenkins-bot: mwconf: no more set $wgHTTPProxy [integration/jenkins] - 10https://gerrit.wikimedia.org/r/261096 (https://phabricator.wikimedia.org/T122449) (owner: 10Hashar) [21:45:25] 10Continuous-Integration-Infrastructure, 6operations: Webproxy on carbon unreachable from labs instances since Dec 24 roughly 1am - https://phabricator.wikimedia.org/T122461#1904999 (10hashar) We have pointed the MediaWiki configuration on CI to a proxy because we had some hosts that had no direct access to i... [21:52:32] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1905000 (10hashar) 5Open>3Resolved a:3hashar Should be fixed now. [21:56:03] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [22:03:56] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, 6Repository-Admins: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1905004 (10Luke081515) [22:07:39] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves - https://phabricator.wikimedia.org/T122449#1905007 (10hashar) So it was potentially a race condition with EventLogging doing a bunch of queries timing out on their own... [22:13:24] 10Continuous-Integration-Config, 10Analytics-EventLogging: EventLogging CI jobs hit meta.wikimedia.org - https://phabricator.wikimedia.org/T122463#1905018 (10hashar) 3NEW a:3hashar [22:17:08] 5Continuous-Integration-Scaling, 6Labs: Nodepool can't SSH to spawn instances due to wmflabs provided instances due ssh algorithms - https://phabricator.wikimedia.org/T102459#1905029 (10hashar) Yup I have filled a bug for Shinken {T121767}. The instance has been deleted as far as I can tell. [22:19:14] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1905033 (10hashar) ``` -h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G) ``` @scfc `5/5`. I wi... [22:42:21] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 3releng-201516-q3, and 5 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#1905039 (10hashar) A lot of people repeatedly stated that GitHub is a dead end, a few things I r... [22:50:55] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.86 ms