[02:37:05] https://integration.wikimedia.org/ci/job/operations-puppet-tox-pep8-jessie/6390/console [02:37:15] 02:30:53 + rsync --archive --compress --contimeout 3 rsync://castor.integration.eqiad.wmflabs:/caches/operations-puppet/production/operations-puppet-tox-pep8-jessie/ /home/jenkins [02:37:16] 02:30:53 rsync: change_dir "/operations-puppet/production/operations-puppet-tox-pep8-jessie" (in caches) failed: No such file or directory (2) [03:07:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #946: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/946/ [04:23:09] Yippee, build fixed! [04:23:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #785: 09FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/785/ [05:24:32] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #761: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/761/ [06:22:08] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2071986 (10Qgil) It is very nice to see that this effort is getting momentum! Thank you very much to all contributors. On the... [06:34:15] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:26:51] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2071986 (10Legoktm) I don't know how many times it needs to be said - it's not a technical problem, it's a social one. Creating... [07:49:21] Project beta-scap-eqiad build #98770: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98770/ [08:05:34] Project beta-scap-eqiad build #98771: 04STILL FAILING in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98771/ [08:09:59] Project beta-scap-eqiad build #98772: 04STILL FAILING in 2 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98772/ [08:17:35] Project beta-scap-eqiad build #98773: 04STILL FAILING in 2 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98773/ [08:20:03] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213431 (10Luke081515) 05Resolved>03Open 10:17:32 08:17:32 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.eqiad.wmflabs::commo... [08:27:23] Project beta-scap-eqiad build #98774: 04STILL FAILING in 2 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98774/ [08:30:00] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2213444 (10Qgil) I think the perspective of generalistic and full time contributors able to follow the activity of channels lik... [08:33:20] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213446 (10hashar) On deployment-tin I have deleted `/srv/mediawiki/php-1.27.0-wmf.13` that might help a bit. [08:35:08] Yippee, build fixed! [08:35:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #944: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/944/ [08:37:40] Project beta-scap-eqiad build #98775: 04STILL FAILING in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98775/ [08:39:43] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213453 (10Luke081515) 10:37 < wmf-insecte> Project beta-scap-eqiad build #98775: STILL FAILING in 2 min 44 sec: https://integration.wikim... [08:42:17] Project beta-scap-eqiad build #98776: 04STILL FAILING in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98776/ [08:43:09] 10Beta-Cluster-Infrastructure: Finish removal of deployment-cache-mobile04 - https://phabricator.wikimedia.org/T130473#2213469 (10hashar) 05Resolved>03Open The instance had a public IP mapped `208.80.155.139` we would want to remove it from wikitech so it can be reused. [08:43:59] 10Beta-Cluster-Infrastructure: Finish removal of deployment-cache-mobile04 - https://phabricator.wikimedia.org/T130473#2213471 (10hashar) p:05Triage>03Normal [08:47:25] Project beta-scap-eqiad build #98777: 04STILL FAILING in 2 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98777/ [08:48:22] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213474 (10Luke081515) p:05Triage>03High [08:52:36] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Jenkins: Use strict mode when running hhvm tests on jenkins - https://phabricator.wikimedia.org/T132270#2193039 (10hashar) At first, both jobs are running exactly the same command but the test suites in MediaWiki core might discard some tests... [08:53:07] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213494 (10hashar) I really mean `/srv/mediawiki-staging/php-1.27.0-wmf.13` [08:54:18] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Bring co-master / fanout capabilities to `deploy` and friends - https://phabricator.wikimedia.org/T121276#2213513 (10Joe) Why shouldn't we just use apache for this? It's easy to set up a specific virtual host on a high port for every repository which is configured via a p... [08:55:26] Project beta-scap-eqiad build #98778: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98778/ [08:59:56] 10Continuous-Integration-Infrastructure, 10Differential, 10Phabricator, 06Project-Admins: Need to create #meta-* projects without formally discussing each one first. - https://phabricator.wikimedia.org/T132888#2213529 (10Aklapper) Mukunda, thanks a lot for taking the time to elaborate and explain this so i... [09:01:24] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 13Patch-For-Review: Send beta cluster Jenkins alerts to betacluster-alert list - https://phabricator.wikimedia.org/T1125#2213532 (10hashar) 05Resolved>03Open ``` Your mail to 'Betacluster-alerts' with the subject beta-scap-eqiad - Build... [09:05:28] Project beta-scap-eqiad build #98779: 04STILL FAILING in 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98779/ [09:08:39] 10Continuous-Integration-Infrastructure, 10Differential, 10Phabricator, 06Project-Admins: Need to create #meta-* projects without formally discussing each one first. - https://phabricator.wikimedia.org/T132888#2213545 (10MarcoAurelio) > I would like to amend the project creation guidelines to explicitly al... [09:11:05] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 13Patch-For-Review: Send beta cluster Jenkins alerts to betacluster-alert list - https://phabricator.wikimedia.org/T1125#2213562 (10hashar) 05Open>03Resolved > To: jenkins-bot@wikimedia.org > Welcome to the Betacluster-alerts@lists.wikimedia... [09:15:27] Project beta-scap-eqiad build #98780: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98780/ [09:23:31] Project beta-scap-eqiad build #98781: 04STILL FAILING in 4 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98781/ [09:24:22] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213661 (10hashar) I have screwed up a bunch of permission on deployment-tin and eventually restored them. [09:27:56] Project beta-scap-eqiad build #98782: 04STILL FAILING in 2 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98782/ [09:38:50] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #783: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/783/ [09:39:51] Yippee, build fixed! [09:39:51] Project beta-scap-eqiad build #98783: 09FIXED in 4 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98783/ [09:43:31] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta-scap-eqiad fails: deployment-mediawiki01 partition is full - https://phabricator.wikimedia.org/T132882#2213692 (10hashar) Fixed again. We really want `/srv` to use the instance extended disk space on all appservers [12:54:08] jzerebecki: around? [12:54:17] addshore: . [12:54:20] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/59058/console [12:54:26] happening on multiple core tests [12:54:38] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #1023: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/1023/ [12:54:54] defiantly not sue to the patches ;) [12:55:54] addshore: git was killed on that workspace while it had a lock in place... [12:56:45] It feels like the lock might be stuck [12:57:32] !log integration-slave-trusty-1014:~$ sudo -i 'rm -f /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/GlobalCssJs/.git/refs/heads/wmf/1.27.0-wmf.11.lock' [12:58:12] ty! [12:59:08] correction: [12:59:10] !log integration-slave-trusty-1014:~$ sudo -i rm -f /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/GlobalCssJs/.git/refs/heads/wmf/1.27.0-wmf.11.lock [13:08:47] jzerebecki: Seems like morebots doesn't like both :P [13:11:04] Luke081515: it needs to be restarted :( or preferably somone changes it so that it does that automatically [13:30:17] (03Draft4) 10Addshore: Relax squizlabs/php_codesniffer version requirement [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) [13:31:52] (03CR) 10Addshore: [C: 04-1] "Please see https://gerrit.wikimedia.org/r/#/c/283966/ & https://gerrit.wikimedia.org/r/#/c/283967/" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/281816 (owner: 10Paladox) [13:33:50] (03CR) 10Jeroen De Dauw: [C: 031] Relax squizlabs/php_codesniffer version requirement [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) (owner: 10Addshore) [13:41:45] (03CR) 10Addshore: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [13:54:48] (03CR) 10Addshore: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [13:55:36] (03CR) 10Addshore: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283384 (https://phabricator.wikimedia.org/T127572) (owner: 10Aashaka) [13:56:00] (03CR) 10Addshore: "Is there a reason we are so strict on the version of phpunit in the requirements?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/281817 (owner: 10Paladox) [13:58:45] (03PS1) 10Addshore: Relax phpunit/phpunit version requirement [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283973 [14:04:00] (03CR) 10Jeroen De Dauw: Relax phpunit/phpunit version requirement (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283973 (owner: 10Addshore) [14:04:52] (03CR) 10Addshore: Relax phpunit/phpunit version requirement (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283973 (owner: 10Addshore) [14:05:39] (03PS2) 10Addshore: Relax phpunit/phpunit version requirement [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283973 [14:07:48] (03CR) 10Jeroen De Dauw: [C: 031] Relax phpunit/phpunit version requirement [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283973 (owner: 10Addshore) [14:09:42] (03CR) 10jenkins-bot: [V: 04-1] Add sniff to check if boolean operator on same line inside 'if condition' [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [14:10:42] (03CR) 10Addshore: "The tests don't pass!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [14:14:18] (03CR) 10Paladox: "But then what happends if they push an update that requires us to fix it on our end. It would cause any repo that updates to this and that" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) (owner: 10Addshore) [14:14:20] (03PS1) 10Luke081515: Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 [14:15:18] (03CR) 10Paladox: [C: 031] Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 (owner: 10Luke081515) [14:15:26] (03CR) 10Aashaka: "I will look into it again." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [14:15:32] (03CR) 10jenkins-bot: [V: 04-1] Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 (owner: 10Luke081515) [14:17:38] (03PS2) 10Luke081515: Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 [14:18:21] (03CR) 10Paladox: [C: 031] Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 (owner: 10Luke081515) [14:18:52] (03CR) 10Addshore: "In theory that will not happen because semver." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) (owner: 10Addshore) [14:21:03] jzerebecki: Maybe you can take a quick look at my patch from above? It's only a very small patch for my new repo ;) [14:21:27] (03CR) 10jenkins-bot: [V: 04-1] Stop reference of string $content as an array [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283384 (https://phabricator.wikimedia.org/T127572) (owner: 10Aashaka) [14:39:59] (03CR) 10JanZerebecki: [C: 032] Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 (owner: 10Luke081515) [14:40:14] jzerebecki: Many thanks :) [14:41:02] (03Merged) 10jenkins-bot: Add php55lint to labs/tools/Luke081515IRCBot [integration/config] - 10https://gerrit.wikimedia.org/r/283976 (owner: 10Luke081515) [14:51:38] (03CR) 10Paladox: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [14:51:43] !log reloading zuul for d9e80bd..113ea21 [14:52:36] Luke081515: you're welcome [15:22:13] 10Beta-Cluster-Infrastructure: Finish removal of deployment-cache-mobile04 - https://phabricator.wikimedia.org/T130473#2214207 (10Krenair) 05Open>03Resolved It was removed in horizon, wikitech no longer handles that stuff [15:29:07] I'd like to migrate integration-slave-trusty-1018 — can someone refresh my memory about how to depool a ci slave? [15:30:05] nm, found it [15:35:13] PROBLEM - Host integration-slave-trusty-1018 is DOWN: CRITICAL - Host Unreachable (10.68.16.240) [15:35:39] (03CR) 10BryanDavis: "My confusion from I9c3a349 is clarified when seeing this change. You want to loosen the requirements on this package but still protect fro" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) (owner: 10Addshore) [15:37:43] (03CR) 10Addshore: "Ahh I did not realise that I had not linked to this patch from that!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283966 (https://phabricator.wikimedia.org/T100544) (owner: 10Addshore) [15:49:41] 10Beta-Cluster-Infrastructure, 10Monitoring, 07Varnish: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#2214282 (10Krenair) nrpe in beta? Do we have icinga/nagios running somewhere able to use that? [15:52:51] 10Beta-Cluster-Infrastructure, 10Monitoring, 07Varnish: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#2214303 (10hashar) We used to have Icinga with generated config files that used nrpe to trigger remote checks. Then we eventually switche... [15:53:49] 10Beta-Cluster-Infrastructure, 10Monitoring, 13Patch-For-Review, 07Shinken: Monitor keyholder on deployment-prep deployment servers - https://phabricator.wikimedia.org/T111064#2214305 (10Krenair) Per @YuviPanda's comments I rewrote it with diamond and it now sends to http://graphite.wmflabs.org/render/?fro... [15:56:22] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 13Patch-For-Review: Send beta cluster Jenkins alerts to betacluster-alert list - https://phabricator.wikimedia.org/T1125#2214307 (10hashar) And I have poked the QA list about it https://lists.wikimedia.org/pipermail/qa/2016-April/002484.html [16:09:20] RECOVERY - Host integration-slave-trusty-1018 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [16:56:29] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2214580 (10Legoktm) >>! In T128371#2213444, @Qgil wrote: > I think the perspective of generalistic and full time contributors a... [17:10:45] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2214616 (10Luke081515) The situation with grrrit-vm at the moment is not optimal, I agree. The original intention was to let pe... [17:15:21] 07Browser-Tests, 06Release-Engineering-Team, 10MobileFrontend: Add helper for detecting if ResourceLoader module (JavaScript) has loaded - https://phabricator.wikimedia.org/T132753#2214631 (10Jdlrobson) p:05Triage>03Low [17:15:30] (03CR) 10Addshore: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [17:23:18] (03CR) 10Paladox: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [17:25:46] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2214684 (10mmodell) @legoktm: That was an honest mistake. I can only offer my apologies that we inconvenienced you. I really d... [17:29:29] 03Scap3, 10scap: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2214705 (10mmodell) a:03mmodell [17:37:09] (03CR) 10Addshore: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [17:43:01] (03CR) 10Paladox: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [17:58:51] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2214876 (10Dereckson) In a perfect world, we would have a general #wikimedia dev channel, without bots, to discuss anything dev... [18:22:01] (03CR) 10Addshore: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [18:26:04] (03PS11) 10Paladox: Update composer to 1.0.0 stable [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) [18:26:15] (03CR) 10Paladox: Update composer to 1.0.0 stable (031 comment) [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [18:28:36] Project beta-scap-eqiad build #98836: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98836/ [18:32:59] Project beta-scap-eqiad build #98837: 04STILL FAILING in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98837/ [18:37:33] Project beta-scap-eqiad build #98838: 04STILL FAILING in 2 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98838/ [18:43:55] Project beta-scap-eqiad build #98839: 04STILL FAILING in 2 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98839/ [18:47:31] !log deployment-mediawiki01: rm /var/log/apache2/*.gz [18:53:01] !log Ran ~bd808/cleanup-var-crap.sh to free a bit of space on deployment-mediawiki01; this instance really needs a bigger disk [18:57:00] (03CR) 10Addshore: [C: 031] "I don't see any major issues with this." [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [18:59:48] Yippee, build fixed! [18:59:49] Project beta-scap-eqiad build #98840: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/98840/ [19:12:38] say, does anybody here know how to tail apache web logs for the beta cluster? [19:18:55] yes [19:19:20] Krenair: ? i just ssh'd into deployment-mediawiki01, but is that comprehensive? [19:19:24] which logs are you looking for? [19:19:37] no, there are two other deployment-mediawiki* machines [19:20:02] Krenair: looking for en.wikipedia.beta.wmflabs.org [19:20:06] 10Continuous-Integration-Infrastructure, 05Gerrit-Migration, 03releng-201516-q4, 10Differential: Spec out needed glue for Differential to Gearman to Nodepool - https://phabricator.wikimedia.org/T130949#2215272 (10JanZerebecki) [19:20:10] 05Gerrit-Migration, 05Gitblit-Deprecate, 06Release-Engineering-Team, 10releng-201516-q3, and 4 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#2215271 (10JanZerebecki) [19:21:28] dr0ptp4kt, it can be served by any of the three servers, just like in prod [19:21:35] but obvious prod has several hundred more machines, we have 3 [19:21:40] Krenair: lol [19:21:50] you're looking for error.log? [19:23:08] 05Gerrit-Migration, 10Differential: Cross-repository gating of changes pre-merge in Differential - https://phabricator.wikimedia.org/T131955#2215277 (10JanZerebecki) [19:23:11] 10Continuous-Integration-Infrastructure, 05Gerrit-Migration, 03releng-201516-q4, 10Differential: Spec out needed glue for Differential to Gearman to Nodepool - https://phabricator.wikimedia.org/T130949#2215276 (10JanZerebecki) [19:23:25] Krenair: looking for the ncsa standard web logs with status codes, request paths, etc. i couldn't seem to find the data in hive and thought, oh, right, that's probably because it doesn't get picked up there [19:23:53] we have hive set up in beta? [19:24:03] Krenair: no, i don't think we do :( [19:24:14] well you certainly won't catch beta cluster in prod hive :) [19:24:36] anyway those are in /var/log/apache2/* on deployment-mediawiki0[1-3] [19:25:06] Krenair: looks like i don't have access? [19:25:40] use sudo [19:26:38] Krenair: word [19:26:48] Krenair: ah, there we go, things tailing [19:27:23] Krenair: yay, looks like i hit on my api.php request, too :) thanks! [19:27:36] np [19:39:07] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend: Add helper to mediawiki_selenium for detecting if ResourceLoader module (JavaScript) has loaded - https://phabricator.wikimedia.org/T132753#2215311 (10hashar) [19:43:49] 10Continuous-Integration-Infrastructure, 05Gerrit-Migration, 03releng-201516-q4, 10Differential: Spec out needed glue for Differential to Gearman to Nodepool - https://phabricator.wikimedia.org/T130949#2215318 (10greg) [19:43:53] 05Gerrit-Migration, 05Gitblit-Deprecate, 06Release-Engineering-Team, 10releng-201516-q3, and 4 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#2215317 (10greg) [19:43:55] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config: Convert puppetmaster sync cronjob to Jenkins job - https://phabricator.wikimedia.org/T73305#2215319 (10hashar) #jenkins is solely for Jenkins itself for job definitions that is #continuous-integration-config [19:44:21] 05Gerrit-Migration, 05Gitblit-Deprecate, 06Release-Engineering-Team, 10releng-201516-q3, and 4 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#1839726 (10greg) Specific blockers aren't needed, that's the whole of #gerrit-migration [19:52:26] 10Beta-Cluster-Infrastructure: Upgrade varnish automatically via puppet in Beta Cluster - https://phabricator.wikimedia.org/T75564#2215334 (10hashar) 05Open>03Resolved a:03hashar The last version is varnish (3.0.6plus-wm8) from 23 Sep 2015 so unattended upgrade hasn't been triggered for that specific packa... [19:54:55] 10Beta-Cluster-Infrastructure: Investigate whether deployment-memc04 instance is still needed - https://phabricator.wikimedia.org/T128178#2215343 (10hashar) And deployment-memc04 has been deleted. [19:56:18] 10Beta-Cluster-Infrastructure, 10MediaWiki-Recent-changes: Special:RecentChanges shows still creation and edits of already deleted pages at dewiki beta - https://phabricator.wikimedia.org/T119559#2215373 (10hashar) 05Open>03Resolved a:03hashar Assuming it got fixed by @krenair comment [19:56:41] 10Beta-Cluster-Infrastructure, 07JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#2215376 (10hashar) [19:57:58] 10Beta-Cluster-Infrastructure, 07JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#576690 (10hashar) 05Open>03declined After two years, there is no champion for this task. I am thus assuming it is not of any specific interest nor it is gett... [19:59:58] 10Beta-Cluster-Infrastructure, 07Puppet: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2215384 (10hashar) The self puppetmaster also tend to auto sign the client certificates. So in theory any labs instance could point to the beta puppet master and... [20:01:11] 10Beta-Cluster-Infrastructure, 07Puppet: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2215387 (10yuvipanda) So if you use resource collection and care about security (you could block access to puppetmaster from outside the project with security gr... [20:18:42] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2071986 (10hashar) The discussion on this task is going way out of the topic of setting up a review hour. Just agree on using e... [20:28:26] 10Beta-Cluster-Infrastructure, 07WorkType-Maintenance: deployment-cache-text04 points to wrong puppetmaster - https://phabricator.wikimedia.org/T120814#2215505 (10hashar) Happened again and filled by @krenair as a more generic task T132689 [20:34:49] 10Beta-Cluster-Infrastructure, 06Labs, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2206880 (10hashar) The `/etc/puppet/puppet.conf` file is generated by concatenating files und... [20:39:14] 10Beta-Cluster-Infrastructure, 06Labs, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2215581 (10hashar) Random trace on deployment-cache-text04 ``` Info: Applying configuration v... [21:08:07] 10Continuous-Integration-Infrastructure: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2215694 (10EBernhardson) After looking into it more i can't suggest using that ppa. It provides a a series of packages prefixed php5.6 and forces you to... [21:22:24] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #241: 04FAILURE in 6 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/241/ [21:24:57] 05Gerrit-Migration, 06Release-Engineering-Team, 10releng-201516-q3, 10ArchCom-RfC, and 2 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#2215769 (10demon) [22:33:19] 10Continuous-Integration-Infrastructure: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2205146 (10greg) https://www.dotdeb.org/about/ [22:59:56] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations, 13Patch-For-Review: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit) - https://phabricator.wikimedia.org/T123718#2216188 (10Dzahn) existing puppet class now runs without errors on furud, systemd unit file is added when on jessie. next is... [23:29:55] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [23:38:55] 07Browser-Tests, 06Collaboration-Team-Interested, 10Flow: More Flow no-JS browser tests - https://phabricator.wikimedia.org/T114729#2216281 (10Mattflaschen)