[00:15:00] 10Continuous-Integration, 3Continuous-Integration-Isolation: Jenkins labs slaves (and later nodepool images) need Gerrit replication for git-clone performance - https://phabricator.wikimedia.org/T87294#1176482 (10Krinkle) [00:15:56] 10Continuous-Integration, 3Continuous-Integration-Isolation: Jenkins labs slaves (and later nodepool images) need Gerrit replication for git-clone performance - https://phabricator.wikimedia.org/T87294#988143 (10Krinkle) Relevant script used by OpenStack for their nodepool instance to set up local git cache:... [00:16:25] greg-g: "Project beta-code-update-eqiad build #50193: FAILURE in 57 sec" does this basically mean i can't expect to test on beta labs? [00:16:31] 10Continuous-Integration, 3Continuous-Integration-Isolation: Nodepool images need Gerrit replication for git-clone performance - https://phabricator.wikimedia.org/T87294#1176486 (10Krinkle) [00:17:25] 10Continuous-Integration, 3Continuous-Integration-Isolation: Isolate contintcloud nova project from the rest of the wmflabs cloud - https://phabricator.wikimedia.org/T86168#1176487 (10Krinkle) [00:17:53] is someone live hacking beta? [00:17:55] 00:13:52 INFO:mwskinpull:cwd: /srv/mediawiki-staging/php-master/skins [00:17:55] 00:13:52 INFO:mwskinpull:running: git pull [00:17:55] 00:13:52 error: The following untracked working tree files would be overwritten by merge: [00:17:58] 10Continuous-Integration, 10Wikimedia-Labs-Infrastructure, 3Continuous-Integration-Isolation: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#1176488 (10Krinkle) [00:17:59] 00:13:52 .gitignore [00:18:01] 00:13:52 Please move or remove them before you can merge. [00:18:04] 00:13:52 Aborting [00:18:18] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/lastFailedBuild/console [00:19:25] greg-g: Hm.. might be caused by something else. I recall Chad working on a patch to switch to using mw-skins and mw-extensions meta repos instead of individual clones [00:19:37] it's been failing with that since 23:33:53 UTC [00:19:39] Which would mean it will overtake the (mostly empty) core directly. [00:19:49] muther [00:19:49] Except that it would then clash on files like README or .gitignore potentially [00:21:10] Krinkle: ideas on how to fix it for now? [00:21:21] Not sure [00:21:37] https://github.com/wikimedia/mediawiki-skins/commit/2e3563c45da857922d1ebaa16b32bfc819785f75 [00:22:00] https://gerrit.wikimedia.org/r/#/c/200307/ [00:22:03] was merged just now [00:22:10] could revert [00:22:18] might still need to reset state manuaully, but it's something [00:22:32] can you try? [00:23:51] 10Continuous-Integration: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit in the short term - https://phabricator.wikimedia.org/T94322#1176520 (10Krinkle) [00:26:17] Krinkle: can I just test by mv'ing the current .gitignore in that dir? [00:26:29] I know that's wrong :( [00:26:52] greg-g: Sure [00:27:07] rm .gitignore in skins/ and then git pull [00:27:53] hmm, well i've got the core bumps ready and can just deploy one to wmf24, its only phase0 so probably not that risky [00:28:05] was hoping to test @beta first though [00:28:37] gjg@deployment-bastion:/srv/mediawiki-staging/php-master/skins$ sudo -u jenkins-deploy git pull [00:28:40] Updating 77e937b..2e3563c [00:28:42] Fast-forward .gitignore | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 .gitignore [00:28:45] gjg@deployment-bastion:/srv/mediawiki-staging/php-master/skins$ [00:30:12] Yippee, build fixed! [00:30:13] Project beta-code-update-eqiad build #50200: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/50200/ [00:30:22] yippee! [00:30:53] woo :) i guess its still a moment to show up on deployment-mediawiki01 though? [00:31:09] !log rm 'd .gitignore in /srv/mediawiki-staging/php-master/skins due to https://gerrit.wikimedia.org/r/#/c/200307/ clashing with a local untracked version [00:31:11] Logged the message, Master [00:31:20] ebernhardson: I just kicked the beta update job, should be live [00:32:29] oh yup, there it is now [00:33:18] can I go drink a cocktail now? [00:33:25] then a beer? [00:33:31] works on beta, woo :) will deploy once the core bumps make it through jenkins (any moment) [00:34:13] well, ~5min if you believe zuul [01:18:35] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1176671 (10Krinkle) a:3hashar [02:24:31] Yippee, build fixed! [02:24:31] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #55: FIXED in 2 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/55/ [02:30:34] (03PS1) 10Krinkle: Enable 'npm' for mwext-Sentry [integration/config] - 10https://gerrit.wikimedia.org/r/201644 [02:31:02] (03CR) 10Krinkle: [C: 032] Enable 'npm' for mwext-Sentry [integration/config] - 10https://gerrit.wikimedia.org/r/201644 (owner: 10Krinkle) [02:32:21] (03Merged) 10jenkins-bot: Enable 'npm' for mwext-Sentry [integration/config] - 10https://gerrit.wikimedia.org/r/201644 (owner: 10Krinkle) [02:36:00] Yippee, build fixed! [02:36:00] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #529: FIXED in 2 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/529/ [02:47:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201644 [02:47:25] Logged the message, Master [03:00:50] (03PS1) 10Krinkle: zuul: Use a relative URL for status.json [integration/docroot] - 10https://gerrit.wikimedia.org/r/201647 [03:00:52] (03PS1) 10Krinkle: zuul: Match zuul_version display logic with upstream [integration/docroot] - 10https://gerrit.wikimedia.org/r/201648 [03:00:54] (03PS1) 10Krinkle: Upgrade Bootstrap to v3.1.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 [03:00:56] (03PS1) 10Krinkle: Page: Implement addStylesheet method [integration/docroot] - 10https://gerrit.wikimedia.org/r/201650 [03:00:58] (03PS1) 10Krinkle: zuul: Merge branch 'master' from openstack-infra [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 [03:01:00] (03PS1) 10Krinkle: zuul: Embed upstream status board instead of forked [integration/docroot] - 10https://gerrit.wikimedia.org/r/201652 [03:01:03] (03CR) 10jenkins-bot: [V: 04-1] zuul: Merge branch 'master' from openstack-infra [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 (owner: 10Krinkle) [03:01:05] (03CR) 10jenkins-bot: [V: 04-1] zuul: Embed upstream status board instead of forked [integration/docroot] - 10https://gerrit.wikimedia.org/r/201652 (owner: 10Krinkle) [03:01:16] (03CR) 10Krinkle: [C: 032] zuul: Use a relative URL for status.json [integration/docroot] - 10https://gerrit.wikimedia.org/r/201647 (owner: 10Krinkle) [03:09:02] (03Merged) 10jenkins-bot: zuul: Use a relative URL for status.json [integration/docroot] - 10https://gerrit.wikimedia.org/r/201647 (owner: 10Krinkle) [03:12:14] !log A bunch of recent builds are mysteriously missing their entry in Jenkins. They showed up in Jenkins's sidebar, but their build url is a 404. E.g. https://integration.wikimedia.org/ci/job/integration-docroot-deploy/105 and https://integration.wikimedia.org/ci/job/jshint/239 [03:12:17] Logged the message, Master [03:18:52] !log The failure started at 03:03 exactly. The newer build metadata exists at /var/lib/jenkins/jobs/:jobname/builds/:nr, but the jobs/*/last*Build symlinks are no longer updated. [03:18:54] Logged the message, Master [03:24:07] 10Continuous-Integration, 7Jenkins: Jenkins: "Warning: Some projects have builds whose timestamps are inconsistent." - https://phabricator.wikimedia.org/T94963#1176838 (10Krinkle) 3NEW [03:24:54] (03CR) 10Krinkle: [C: 032] zuul: Match zuul_version display logic with upstream [integration/docroot] - 10https://gerrit.wikimedia.org/r/201648 (owner: 10Krinkle) [03:24:56] (03Merged) 10jenkins-bot: zuul: Match zuul_version display logic with upstream [integration/docroot] - 10https://gerrit.wikimedia.org/r/201648 (owner: 10Krinkle) [03:26:20] !log Reloading Jenkins configuration from disk to mitigate [03:26:23] Logged the message, Master [03:40:15] 10Continuous-Integration, 7Jenkins: Jenkins: "Warning: Some projects have builds whose timestamps are inconsistent." - https://phabricator.wikimedia.org/T94963#1176877 (10Krinkle) Per upstream Jenkins documentation, I've moved them out of the way to the backup folder so that the warning goes away. Could be inv... [03:41:06] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:42:25] !log Reloading Jenking config repaired the broken references. Build urls are now resolving again. [03:42:28] Logged the message, Master [03:44:00] !log References to past hour of builds have been restored. But Jenkins is still enable to make new references properly. New builds are 404'ing the same way. [03:44:03] Logged the message, Master [03:44:41] !log *unable [03:44:44] Logged the message, Master [03:44:55] :) [03:51:02] !log Jenkins is unable to re-establish Gearman connection. Have to force restart Jenkins master. [03:51:05] Logged the message, Master [03:52:07] wtf [03:53:41] I don't know where this is coming from. [03:54:12] It wouldn't re-enable Gearman. [03:54:16] greg-g: https://wikitech.wikimedia.org/wiki/Release_Engineering/Argh [03:54:25] Just updated it to get a sense of the error rate [03:55:16] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:55:19] (03CR) 10jenkins-bot: [V: 04-1] Upgrade Bootstrap to v3.1.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:55:41] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:56:19] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:56:28] Krinkle: ughhh [03:57:47] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [03:58:52] !log Jobs were throwing NOT_RECOGNISED. Relaunched Gearman. Jobs are now happy again. [03:58:55] Logged the message, Master [03:59:34] (03CR) 10Krinkle: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [04:00:13] I see a topic for the next CI weekly :) [04:02:19] (03PS2) 10Krinkle: zuul: Embed upstream status board instead of forked [integration/docroot] - 10https://gerrit.wikimedia.org/r/201652 [04:02:21] (03PS2) 10Krinkle: zuul: Merge branch 'master' from openstack-infra [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 [04:03:14] (03CR) 10Krinkle: [C: 032] "Needed for compat with upstream" [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [04:03:16] (03Merged) 10jenkins-bot: Upgrade Bootstrap to v3.1.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/201649 (owner: 10Krinkle) [04:03:20] (03CR) 10Krinkle: [C: 032] Page: Implement addStylesheet method [integration/docroot] - 10https://gerrit.wikimedia.org/r/201650 (owner: 10Krinkle) [04:03:23] (03Merged) 10jenkins-bot: Page: Implement addStylesheet method [integration/docroot] - 10https://gerrit.wikimedia.org/r/201650 (owner: 10Krinkle) [04:03:27] (03CR) 10Krinkle: [C: 032] zuul: Merge branch 'master' from openstack-infra [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 (owner: 10Krinkle) [04:03:29] (03Merged) 10jenkins-bot: zuul: Merge branch 'master' from openstack-infra [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 (owner: 10Krinkle) [04:03:38] (03CR) 10Krinkle: [C: 032] zuul: Embed upstream status board instead of forked [integration/docroot] - 10https://gerrit.wikimedia.org/r/201652 (owner: 10Krinkle) [04:03:40] (03Merged) 10jenkins-bot: zuul: Embed upstream status board instead of forked [integration/docroot] - 10https://gerrit.wikimedia.org/r/201652 (owner: 10Krinkle) [04:09:41] (03PS1) 10Krinkle: integration/default: Remove broken status image for old mw-regression jobs [integration/docroot] - 10https://gerrit.wikimedia.org/r/201657 [04:09:56] (03CR) 10Krinkle: [C: 032] integration/default: Remove broken status image for old mw-regression jobs [integration/docroot] - 10https://gerrit.wikimedia.org/r/201657 (owner: 10Krinkle) [04:09:58] (03Merged) 10jenkins-bot: integration/default: Remove broken status image for old mw-regression jobs [integration/docroot] - 10https://gerrit.wikimedia.org/r/201657 (owner: 10Krinkle) [04:36:28] Yippee, build fixed! [04:36:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #397: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/397/ [04:44:48] (03PS7) 10Mattflaschen: Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD to GettingStarted browser test [integration/config] - 10https://gerrit.wikimedia.org/r/194749 (https://phabricator.wikimedia.org/T91220) [04:45:13] (03CR) 10Mattflaschen: "What are the blockers to getting this merged?" [integration/config] - 10https://gerrit.wikimedia.org/r/194749 (https://phabricator.wikimedia.org/T91220) (owner: 10Mattflaschen) [04:45:14] Yippee, build fixed! [04:45:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #402: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/402/ [04:45:33] (03CR) 10Mattflaschen: [C: 031] Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD to GettingStarted browser test [integration/config] - 10https://gerrit.wikimedia.org/r/194749 (https://phabricator.wikimedia.org/T91220) (owner: 10Mattflaschen) [04:46:20] 10Continuous-Integration, 6Collaboration-Team, 10MediaWiki-extensions-GettingStarted, 7Blocked-on-Continuous-Integration, and 2 others: Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD in on Jenkins so GettingStarted browser tests pass - https://phabricator.wikimedia.org/T91220#1176905 (10Mattflaschen) This remains... [05:11:42] (03PS3) 10Legoktm: Move extensions to use generic jshint & jsonlint jobs part 3 [integration/config] - 10https://gerrit.wikimedia.org/r/200744 [05:16:22] 10Continuous-Integration, 7Regression, 7Upstream: ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint' - https://phabricator.wikimedia.org/T93321#1176989 (10Krinkle) [05:28:27] 10Continuous-Integration, 7I18n, 5Patch-For-Review, 7Pywikibot-i18n: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1177011 (10Nemo_bis) [05:30:14] 10Continuous-Integration, 7I18n, 5Patch-For-Review, 7Pywikibot-i18n: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1177017 (10Legoktm) What's still missing exactly? There's a jsonlint job that verifies all JSON files have valid syntax,... [05:31:33] 10Continuous-Integration, 7I18n, 5Patch-For-Review, 7Pywikibot-i18n: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1177018 (10Legoktm) Oh, I read too fast. I think you're refering to. >>! In T85335#1147403, @jayvdb wrote: > We also ne... [05:32:08] 10Continuous-Integration, 7I18n, 5Patch-For-Review, 7Pywikibot-i18n: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1177020 (10jayvdb) @legoktm, new JS files (e.g. https://gerrit.wikimedia.org/r/#/c/196706/) are not checked by banana wh... [05:36:50] (03CR) 10Legoktm: [C: 032] Move extensions to use generic jshint & jsonlint jobs part 3 [integration/config] - 10https://gerrit.wikimedia.org/r/200744 (owner: 10Legoktm) [05:38:23] (03Merged) 10jenkins-bot: Move extensions to use generic jshint & jsonlint jobs part 3 [integration/config] - 10https://gerrit.wikimedia.org/r/200744 (owner: 10Legoktm) [05:41:11] 10Continuous-Integration, 10pywikibot-core, 5Patch-For-Review: run pep8 and pep257 for pywikibot/core - https://phabricator.wikimedia.org/T87169#1177030 (10Legoktm) My hacky idea is to have a pywikibot-core-tox-flake8 job that only runs if "tox.ini" is *not* touched for non-whitelisted users. Whitelisted use... [05:42:03] !log deploying https://gerrit.wikimedia.org/r/200744 [05:42:06] Logged the message, Master [05:49:15] there are now 7 voting mwext-*-jslint jobs left [06:02:13] (03PS1) 10Legoktm: Move extensions to use generic jshint & jsonlint jobs part 4 [integration/config] - 10https://gerrit.wikimedia.org/r/201669 [06:02:46] (03CR) 10jenkins-bot: [V: 04-1] Move extensions to use generic jshint & jsonlint jobs part 4 [integration/config] - 10https://gerrit.wikimedia.org/r/201669 (owner: 10Legoktm) [06:04:54] (03PS2) 10Legoktm: Move extensions to use generic jshint & jsonlint jobs part 4 [integration/config] - 10https://gerrit.wikimedia.org/r/201669 [06:08:51] Krinkle: what do you think about creating a generic 'jshint-nonvoting' job to use in extensions where mwext-*-jslint is currently nonvoting? [06:09:41] there are 364 extensions with it non voting right now [06:09:47] that many? [06:10:07] legoktm: Maybe find out how many of them don't even have a .jshintrc file and remove those completely. [06:10:27] I should've cleaned that up years ago, but we kept adding it to the default whenever an extension got added [06:10:41] Without a .jshintrc it's just spinning wheels nothign nothing useful. [06:11:33] that's probably most/all of them? [06:13:48] legoktm: I'd say, for any that are deployed, if there's no jshintrc file, assign the task to me and I'll add one and make it passing. For the others, remove the job. [06:14:01] ok [06:14:16] It's gotta have some proactiveness from the repo maintainers if they want it [06:14:42] OK. I now have a draft for Chrome/Karma, MySQL, Zuul dashboard and then this? [06:14:46] (announcements) [06:15:06] Will hopefully send one or two of them tomorrow [06:15:07] o/ [06:15:09] night [06:15:16] night! [07:02:15] RECOVERY - Puppet staleness on deployment-bastion is OK: OK: Less than 1.00% above the threshold [3600.0] [07:07:41] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-03-24, 5Patch-For-Review: Switch wikidata qunit jobs from qunit to qunit-karma - https://phabricator.wikimedia.org/T94393#1177208 (10Krinkle) [07:07:43] 10Continuous-Integration, 10VisualEditor, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#1177207 (10Krinkle) [07:18:44] 10Continuous-Integration, 10pywikibot-core, 5Patch-For-Review: run pep8 and pep257 for pywikibot/core - https://phabricator.wikimedia.org/T87169#1177219 (10jayvdb) @legoktm, that sounds much better than the current situation - tox.ini changes are very rare by non-whitelisted users. package `flake8-docstring... [07:37:29] (03PS1) 10Krinkle: Add support for multiple virtual hosts to doc publisher [integration/config] - 10https://gerrit.wikimedia.org/r/201677 (https://phabricator.wikimedia.org/T93558) [07:53:05] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:45] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:56] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 49290 bytes in 0.581 second response time [07:58:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 30078 bytes in 0.685 second response time [07:59:11] good morning [08:01:01] hashar: sorry, I will be 5-10 minutes late for the meeting :( [08:05:25] zeljkof: sure [08:05:30] zeljkof: looking at some prod issue [08:06:12] (03CR) 10Krinkle: [C: 032] "Deployed publish-on-gallium and *-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/201677 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [08:08:29] hashar: ready, joining hangout [08:10:44] (03Merged) 10jenkins-bot: Add support for multiple virtual hosts to doc publisher [integration/config] - 10https://gerrit.wikimedia.org/r/201677 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [08:27:51] (03CR) 10Jhernandez: [C: 031] Docs jobs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/191046 (https://phabricator.wikimedia.org/T74794) (owner: 10Hashar) [08:37:56] (03PS1) 10Krinkle: dox: Fix yaml to make line break real [integration/config] - 10https://gerrit.wikimedia.org/r/201679 [08:37:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:24] (03PS1) 10Krinkle: gitignore: Clean up to match prod so there's no dirty git status [integration/docroot] - 10https://gerrit.wikimedia.org/r/201681 [08:42:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 49070 bytes in 0.541 second response time [08:48:21] Krinkle|detached: nice work going on the Zuul status page :-D [08:48:38] Krinkle|detached: bikeshedd: the colors on the progress bars are terrible for my poor color blind eyes hehe [08:52:24] (03PS2) 10Krinkle: dox: Fix yaml to make line break real [integration/config] - 10https://gerrit.wikimedia.org/r/201679 [08:52:40] (03CR) 10Krinkle: [C: 032] gitignore: Clean up to match prod so there's no dirty git status [integration/docroot] - 10https://gerrit.wikimedia.org/r/201681 (owner: 10Krinkle) [08:52:42] (03Merged) 10jenkins-bot: gitignore: Clean up to match prod so there's no dirty git status [integration/docroot] - 10https://gerrit.wikimedia.org/r/201681 (owner: 10Krinkle) [09:04:45] zeljkof: https://selenium.googlecode.com/svn/trunk/docs/api/rb/Selenium/WebDriver/Support/AbstractEventListener.html [09:04:45] :D [09:05:16] hashar: http://seleniumhq.github.io/selenium/docs/api/rb/Selenium/WebDriver/Logs.html [09:06:18] (03CR) 10Krinkle: [C: 032] "Deployed. Publishing is now working again." [integration/config] - 10https://gerrit.wikimedia.org/r/201679 (owner: 10Krinkle) [09:06:57] (03CR) 10Krinkle: "The url is now also displayed in the first build https://integration.wikimedia.org/ci/job/mwext-VisualEditor-jsduck-publish/130/console" [integration/config] - 10https://gerrit.wikimedia.org/r/201677 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [09:08:24] zeljkof: and dont forget to shop candies and chocolates eggs for kids ! [09:08:35] hashar: :) [09:10:48] (03Merged) 10jenkins-bot: dox: Fix yaml to make line break real [integration/config] - 10https://gerrit.wikimedia.org/r/201679 (owner: 10Krinkle) [09:12:04] zeljkof: dont forget to fill a bug for SyntaxHiglight-GeSHi extension [09:12:20] which when using 'yaml' inserts a space before comments :( [09:12:32] I cant remember how we use Geshi [09:12:36] it is an upstream repo [09:12:45] that used to be hosted on a sourceforge svn. Reedy had commit right on it [09:12:48] rights [09:13:33] hashar: will do [09:14:04] most probably someone from the community will spot it and fix it :D [09:30:41] (03CR) 10Hashar: "Thanks a ton for that update. The subway map like thing is really useful." [integration/docroot] - 10https://gerrit.wikimedia.org/r/201651 (owner: 10Krinkle) [09:48:11] geshi is basically abandoned by upstream [09:56:33] thedj: good to know [09:56:50] and it migrated to git https://github.com/GeSHi/ [10:00:16] hashar: We have several outstanding pull requests there, and those are merged locally for us. [10:00:38] Reedy no longer has commit access for them either as far as I'm aware [10:01:22] maybe we could reach out to upstream [10:01:23] I've suggested we fork them and pull from our own wikimedia/GeSHi but it hasn't happend so far. [10:01:29] and propose to take over the project under Wikimedia umbrella [10:01:33] hashar: Reedy tried a lot in the past. [10:01:44] so lets fork it [10:01:53] either on github or on gerrit [10:03:14] thedj: do you want to take the lead on it ? :) [10:03:32] we can probably loop in a few people such as the one for the Librarization project [10:03:39] or find out people interested in maintaining it [10:03:43] thedj: ori and I are strongly considering dropping Geshi entirely [10:03:49] then from there pick a choice between a github repo or Gerrit [10:03:50] It's not worth our time to fix it [10:04:16] Krinkle: thanks for the Zuul status page update :=D [10:04:18] yeah, that's why i haven't so far [10:04:26] https://phabricator.wikimedia.org/T85794 [10:04:31] https://phabricator.wikimedia.org/T85796 [10:04:32] etc. [10:04:35] Krinkle: bikeshedding : the colors of the progress bars are terrible hehe [10:04:49] hashar: Yeah, it was a lot of work. Worked with upstream to make it happen last few days [10:04:56] well done! [10:05:13] hashar: I did another commit after it that separates it entirely, so it's no longer a fork. [10:05:22] https://review.openstack.org/#/q/owner:%22Timo+Tijhof%22+status:open,n,z [10:05:25] just found out the progress bar red/green/blue represents the hidden jobs being run huhu [10:05:28] I had some bikesheds too [10:05:34] Patches upstream welcome :) [10:05:50] yeah it is good to know we reconcialite with upstream [10:05:59] so you essentially dropped most of the rewrite you did ? [10:06:11] No [10:06:24] My rewrite is what they used [10:06:28] I upstreamed it remember :) [10:06:42] They didnt have it as part of the zuul repo before [10:07:21] They're working on making their branded dashboard use this one as well, similar to what I did with ours. There's an unbranded basic html file present by default, but all code is generic so it can be re-used in another enviornment. [10:07:27] Like our mini PHP framework [10:07:35] Or their status.openstack.org/zuul (which doesn't use it yet) [10:08:05] They made some patches on top of it, but the basic code structure is still there from my rewrite. But yeah, they rewrote a lot of it. Which is fine :) [10:08:25] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1177395 (10hashar) [10:08:57] hashar: How did you recreate puppetmaster? Deleting and then re-creating? What happens if it's temporary not existing? [10:09:07] I was scared to do it as I wasn't sure what modules and configuration it needs [10:11:56] !sal [10:11:56] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:13:58] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1177399 (10hashar) I have recreated the instance to benefit from a new partitioning scheme which has an extended /var {T87484}. We indeed want to downgraded per @B... [10:14:34] 10Continuous-Integration, 5Patch-For-Review: Recreate integration-puppetmaster with new image (/var/ is too small) - https://phabricator.wikimedia.org/T87484#1177402 (10hashar) The new instance has been recreated as a Trusty one which comes with a different puppet version not supported by ops. So we want to do... [10:14:44] Krinkle: so yeah [10:14:47] basically delete the instance [10:14:54] recreated it with the SAME name [10:14:57] and a Precise image [10:15:08] run puppet on it several time [10:15:13] then make it a puppetmaster [10:15:30] there is a lot of mess ensueing because the certs are changed and need to be regenerated [10:15:40] though potentially one could save all the certs from the current instance [10:15:44] I have no idea where they are though [10:16:14] https://phabricator.wikimedia.org/T87484#1081655 [10:16:32] (03PS1) 10Krinkle: Publish oojs code coverage to integration.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) [10:16:55] (03CR) 10Krinkle: [C: 04-1] "Requires minor clean up (I5dbd155ac9) in oojs/core." [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [10:17:19] hashar: we have autosigning btw. [10:17:33] hashar: I noticed you logged that you had to do signing on a new instance, that shouldn't be needed. [10:18:07] might not work for existing instances though, but at least it works for new ones. [10:19:59] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1177410 (10hashar) In short: * delete integration-puppetmaster * recreate it with the same name * once booted run puppet a bunch of time to complete the setup * on... [10:20:18] Krinkle: yeah that was introduced meanwhile [10:20:25] the thing is that the client instances have a cert already [10:20:28] I enabled it a few days ago [10:20:30] not sure how it will work [10:20:33] maybe it will [10:20:45] want me to do the downgrade? [10:20:47] Worked fine on all instances I cerated [10:20:52] Yeah, that'd be cool [10:20:54] I'm too scared. [10:20:55] :D [10:21:03] brb in a few hours [10:21:04] o/ [10:21:07] doing so [10:21:10] sleep well! [10:21:37] !log downgrading integration-puppetmaster from Trusty to Precise https://phabricator.wikimedia.org/T94927 [10:21:41] Logged the message, Master [10:22:21] !log Created instance i-00000a4a with image "ubuntu-12.04-precise" and hostname i-00000a4a.eqiad.wmflabs. [10:22:23] Logged the message, Master [10:22:54] * hashar whistles while some computers work [10:26:42] 6Release-Engineering: Design a Test-Driven Development (TDD) survey - https://phabricator.wikimedia.org/T94472#1177415 (10zeljkofilipin) >>! In T94472#1175305, @Jdlrobson wrote: > @zeljkofilipin could be a good question for the survey? "Does Wikimedia have a QA team?" :) It is easy to answer, if you just take a... [10:26:45] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [10:31:37] PROBLEM - Puppet failure on integration-slave1001 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [10:31:47] PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:33:06] PROBLEM - Puppet failure on integration-slave1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:36:30] RECOVERY - Host integration-puppetmaster is UP: PING OK - Packet loss = 0%, RTA = 1.52 ms [10:37:21] !log disabled some hiera configuration related to puppetmaster. [10:37:23] Logged the message, Master [10:38:04] PROBLEM - Puppet failure on integration-slave1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:38:34] PROBLEM - Puppet failure on integration-slave-precise-1011 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [10:38:52] PROBLEM - Puppet failure on integration-slave-precise-1012 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [10:40:24] Connect error: No such file or directory (uri="ldap://ldap-codfw.wikimedia.org:389") [10:40:26] yehhhhhh [10:40:41] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:41:22] PROBLEM - Puppet failure on integration-slave1402 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [10:42:46] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.17.130) [10:43:07] PROBLEM - Puppet failure on integration-slave1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:45:18] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:47:57] PROBLEM - Puppet failure on integration-slave1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:48:52] PROBLEM - Puppet failure on integration-slave1004 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [10:56:24] RECOVERY - Host integration-puppetmaster is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms [11:23:09] RECOVERY - Puppet failure on integration-slave1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:23:22] !log rebooting integration-publisher : cant ssh to it [11:23:24] Logged the message, Master [11:26:29] RECOVERY - Puppet failure on integration-slave1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:30] RECOVERY - Puppet failure on integration-slave1402 is OK: OK: Less than 1.00% above the threshold [0.0] [11:27:11] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1177479 (10hashar) 5Open>3Resolved The integration project has some hiera data to have the instance default to point to a puppet master named integration-puppet... [11:27:12] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1177481 (10hashar) [11:28:02] RECOVERY - Puppet failure on integration-slave1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:28:16] RECOVERY - Puppet failure on integration-slave1403 is OK: OK: Less than 1.00% above the threshold [0.0] [11:30:14] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [11:30:24] !log integration-puppetmaster migrated down to Precise [11:30:27] Logged the message, Master [11:42:05] PROBLEM - Host integration-slave-precise-1011 is DOWN: CRITICAL - Host Unreachable (10.68.16.72) [11:42:11] !log recreating integration-slave-precise-1011 stalled with a puppet oddity related to Package['gdb'] defined twice {{bug|T94917}} [11:42:14] Logged the message, Master [11:49:18] !log made integration-puppetmaster to self update its puppet clone [11:49:21] Logged the message, Master [11:55:33] 10Continuous-Integration, 6operations, 5Patch-For-Review, 7Puppet, 7Regression: Puppet: "Package[git-core] is already declared in file modules/authdns/manifests/scripts.pp" - https://phabricator.wikimedia.org/T94921#1177531 (10hashar) Out of the four precise instances Timo created, only one has the probl... [11:58:09] !log Cherry picked a couple patches to fix puppet Package[] definitions issues [11:58:11] Logged the message, Master [11:58:33] !log Applied role::ci::slave::labs on integration-slave-precise-101[1-4] that Timo created earlier [11:58:36] Logged the message, Master [11:58:58] RECOVERY - Host integration-slave-precise-1011 is UP: PING OK - Packet loss = 0%, RTA = 0.83 ms [12:01:10] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:04:46] 10Continuous-Integration: Remove Junit plugin from Jenkins installation - https://phabricator.wikimedia.org/T94685#1177547 (10hashar) p:5Triage>3Low [12:04:59] 10Continuous-Integration, 7Browser-Tests: Browser test jobs should use xUnit publisher instead of Junit - https://phabricator.wikimedia.org/T94684#1177548 (10hashar) p:5Triage>3Normal [12:07:40] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1177552 (10hashar) I have recreated the integration-puppetmaster as Precise {T94927} I have applied the puppet patches for: * {T94921} * {T94917} Puppet is finishing up the installation of `role::ci::slave::la... [12:08:41] 10Continuous-Integration, 6operations, 7Puppet, 7Regression: Puppet: "Could not find class role::ci::slave::labs" - https://phabricator.wikimedia.org/T94925#1177560 (10hashar) 5Open>3Resolved No idea what happened, maybe it was related to the puppetmaster being Trusty? [12:08:43] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1177562 (10hashar) [12:09:24] 10Continuous-Integration, 10Wikimedia-Labs-General, 6operations: role::puppet::self broken on new labs instances - https://phabricator.wikimedia.org/T94834#1177565 (10hashar) [12:16:55] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:19:23] !log enabled puppetmaster::autosigner on integration-puppetmaster [12:19:25] Logged the message, Master [12:24:56] (03PS1) 10Krinkle: zuul: Include jquery-visibility.js [integration/docroot] - 10https://gerrit.wikimedia.org/r/201699 [12:25:57] (03PS2) 10Krinkle: zuul: Include jquery-visibility.js [integration/docroot] - 10https://gerrit.wikimedia.org/r/201699 [12:26:02] (03CR) 10Krinkle: [C: 032] zuul: Include jquery-visibility.js [integration/docroot] - 10https://gerrit.wikimedia.org/r/201699 (owner: 10Krinkle) [12:26:56] 10Continuous-Integration, 10Wikimedia-Labs-General, 6operations: role::puppet::self broken on new labs instances - https://phabricator.wikimedia.org/T94834#1177607 (10hashar) I had an instance suffering of the issue, I had to recreate it. I can confirm puppet runs just fine now. Thank you! [12:27:59] !log integration-slave-jessie-1001 applying role::ci::slave::labs::common to pool it as a very basic Jenkins slave [12:28:01] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 7Documentation: Document how to tag extensions in git - https://phabricator.wikimedia.org/T94412#1177609 (10Aklapper) Apart from the Git heroes only #release-engineering comes to my mind... [12:28:01] Logged the message, Master [12:28:10] (03CR) 10Zfilipin: [C: 031] Browser proxy support for Firefox/Chrome/Phantomjs [selenium] - 10https://gerrit.wikimedia.org/r/201492 (https://phabricator.wikimedia.org/T71725) (owner: 10Dduvall) [12:29:11] zeljkof: just +2 it I guess [12:29:20] zeljkof: unlikely anyone else has the skill to review this change [12:29:28] hashar: true, will do [12:29:43] (03CR) 10Zfilipin: [C: 032] Browser proxy support for Firefox/Chrome/Phantomjs [selenium] - 10https://gerrit.wikimedia.org/r/201492 (https://phabricator.wikimedia.org/T71725) (owner: 10Dduvall) [12:30:27] 6Release-Engineering, 10VisualEditor, 7Browser-Tests, 5Patch-For-Review: Create VisualEditor tests targeting the older version of browsers (Chrome and Firefox for now) for better backward compatibility - https://phabricator.wikimedia.org/T90678#1177614 (10zeljkofilipin) No reply in a week, I am assuming no... [12:30:48] 6Release-Engineering, 10VisualEditor, 7Browser-Tests, 5Patch-For-Review: Create VisualEditor tests targeting the older version of browsers (Chrome and Firefox for now) for better backward compatibility - https://phabricator.wikimedia.org/T90678#1177615 (10zeljkofilipin) 5Open>3declined [12:31:12] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [12:35:15] !log Switching Jessie slave from role::ci::slave::labs::common to role::ci::slave::labs which will bring a whole lot of packages and break [12:35:18] Logged the message, Master [12:35:30] (03Merged) 10jenkins-bot: zuul: Include jquery-visibility.js [integration/docroot] - 10https://gerrit.wikimedia.org/r/201699 (owner: 10Krinkle) [12:35:32] (03Merged) 10jenkins-bot: Browser proxy support for Firefox/Chrome/Phantomjs [selenium] - 10https://gerrit.wikimedia.org/r/201492 (https://phabricator.wikimedia.org/T71725) (owner: 10Dduvall) [12:35:34] 10Continuous-Integration: Create CI slaves using Debian Jessie for debian-glue script - https://phabricator.wikimedia.org/T94836#1177639 (10hashar) After some madness with puppet and OpenStack manager, I got the first Jessie CI slave pooled in Jenkins: https://integration.wikimedia.org/ci/computer/integration-sl... [12:41:50] RECOVERY - Puppet failure on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:38] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [12:56:17] !log installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-1011.eqiad.wmflabs [12:56:20] Logged the message, Master [12:56:25] !log installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-101* instances [12:56:27] Logged the message, Master [12:57:10] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [12:59:11] 10Continuous-Integration: Jessie has no install candidate for openjdk-6-jdk - https://phabricator.wikimedia.org/T94999#1177680 (10hashar) 3NEW [13:02:38] 10Continuous-Integration: Create CI slaves using Debian Jessie - https://phabricator.wikimedia.org/T94836#1177687 (10hashar) [13:03:59] 10Continuous-Integration: Jessie has no install candidate for firefox - https://phabricator.wikimedia.org/T95000#1177691 (10hashar) 3NEW [13:06:17] 10Continuous-Integration: Update puppet for packages having different names in Jessie - https://phabricator.wikimedia.org/T95000#1177697 (10hashar) [13:07:11] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:25] 10Continuous-Integration, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177707 (10hashar) 3NEW [13:16:20] 10Continuous-Integration: Jessie cant use xvfb service because provider upstart is not available - https://phabricator.wikimedia.org/T95003#1177720 (10hashar) 3NEW [13:16:33] (03PS4) 10Zfilipin: Make browser tests version '' by default [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:23:27] (03CR) 10Zfilipin: "Works fine for Firefox:" [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:23:37] 10Continuous-Integration, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177746 (10hashar) [13:23:38] 10Continuous-Integration: Create CI slaves using Debian Jessie - https://phabricator.wikimedia.org/T94836#1177745 (10hashar) [13:25:10] 10Continuous-Integration, 5Patch-For-Review: Jessie has no install candidate for openjdk-6-jdk - https://phabricator.wikimedia.org/T94999#1177747 (10hashar) a:3hashar [13:27:25] (03CR) 10Zfilipin: "Works fine for Chrome:" [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:33:28] (03CR) 10Zfilipin: "Works fine for Safari:" [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:35:37] 10Continuous-Integration: Build Debian package jenkins-debian-glue for Jessie - https://phabricator.wikimedia.org/T95006#1177778 (10hashar) 3NEW [13:35:46] 10Continuous-Integration, 6operations: Build Debian package jenkins-debian-glue for Jessie - https://phabricator.wikimedia.org/T95006#1177778 (10hashar) [13:36:35] (03CR) 10Zfilipin: "Works fine for IE:" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:36:56] zeljkof: spammmmm :) [13:37:03] good to see you are testing it [13:37:33] hashar: it is not a minor change, I wanted to make sure it does not break all browser test jobs :) [13:37:54] I am not sure why I did not test it before, it got lost in my gerrit queue [13:42:17] (03CR) 10Zfilipin: "Works fine for android:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:44:46] (03CR) 10Zfilipin: "Works fine for phantomjs:" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:45:06] 10Continuous-Integration, 6operations: Build Debian package ruby-jsduck for Jessie - https://phabricator.wikimedia.org/T95008#1177814 (10hashar) 3NEW [13:47:44] Yippee, build fixed! [13:47:44] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce build #25: FIXED in 1 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce/25/ [13:48:16] (03CR) 10Zfilipin: "It even fixed iphone job!!!!! :)" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:48:54] (03PS5) 10Zfilipin: Make browser tests version '' by default [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:50:05] (03PS6) 10Zfilipin: Make browser tests version '' (empty string) by default [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:50:19] (03CR) 10Zfilipin: [C: 032] "Deploying all browsertests* jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:50:57] hashar: you have even fixed iphone job!!!!111oneoneone https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce/ [13:53:07] 10Continuous-Integration, 10MediaWiki-extensions-CentralNotice, 7Browser-Tests: Fix failing CentralNotice browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94151#1177834 (10zeljkofilipin) Fixed by https://gerrit.wikimedia.org/r/#/c/200512/ [13:53:16] 10Continuous-Integration, 7Browser-Tests, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1177837 (10zeljkofilipin) [13:53:17] 10Continuous-Integration, 10MediaWiki-extensions-CentralNotice, 7Browser-Tests: Fix failing CentralNotice browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94151#1177835 (10zeljkofilipin) 5Open>3Resolved [13:54:46] (03Merged) 10jenkins-bot: Make browser tests version '' (empty string) by default [integration/config] - 10https://gerrit.wikimedia.org/r/200512 (owner: 10Hashar) [13:58:52] 6Release-Engineering, 6Collaboration-Team, 10Flow, 7Browser-Tests: Investigate updating browser versions in Jenkins builds. - https://phabricator.wikimedia.org/T92005#1177860 (10zeljkofilipin) 5Open>3Resolved [13:59:59] 10Continuous-Integration, 6Collaboration-Team, 10Flow, 7Browser-Tests, 5Patch-For-Review: Fix failed Flow browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94153#1177861 (10zeljkofilipin) [14:00:18] 6Release-Engineering, 6Collaboration-Team, 10Flow, 7Browser-Tests: Investigate updating browser versions in Jenkins builds. - https://phabricator.wikimedia.org/T92005#1101350 (10zeljkofilipin) All browsertests Jenkins jobs are updated to the latest versions supported by Sauce labs: https://gerrit.wikimedia... [14:07:47] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #576: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/576/ [14:13:52] 10Continuous-Integration, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177880 (10hashar) [14:17:36] 10Continuous-Integration, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177889 (10hashar) [14:34:55] Yippee, build fixed! [14:34:55] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #460: FIXED in 8 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/460/ [14:39:46] 6Release-Engineering, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 7Browser-Tests: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1177915 (10zeljkofilipin) [14:39:59] 10Continuous-Integration, 10Wikidata, 7Browser-Tests: Fix failed Wikidata browsertests Jenkins job - https://phabricator.wikimedia.org/T94163#1177918 (10zeljkofilipin) [14:40:00] 6Release-Engineering, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 7Browser-Tests: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10zeljkofilipin) [14:40:11] 10Continuous-Integration, 6operations, 7Blocked-on-Operations: Build Debian package ruby-jsduck for Jessie - https://phabricator.wikimedia.org/T95008#1177814 (10hashar) [14:40:53] 10Continuous-Integration, 7Browser-Tests, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1177923 (10zeljkofilipin) [14:40:54] 6Release-Engineering, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 7Browser-Tests: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10zeljkofilipin) [14:41:31] 10Continuous-Integration, 6operations: Build Debian package jenkins-debian-glue for Jessie - https://phabricator.wikimedia.org/T95006#1177925 (10hashar) a:3hashar [14:41:38] 10Continuous-Integration, 6operations: Build Debian package jenkins-debian-glue for Jessie - https://phabricator.wikimedia.org/T95006#1177778 (10hashar) I have build the package using a cowbuilder image of jessie-wikimedia. Result is published on http://people.wikimedia.org/~hashar/debs/jenkins-debian-glue/... [14:43:39] zeljkof: so VERSION='' fix a bunch of browser tests ? [14:44:33] 10Continuous-Integration, 10MediaWiki-extensions-CentralNotice, 7Browser-Tests: Fix failing CentralNotice browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94151#1177931 (10AndyRussG) Thanks!! Looks great :) [14:45:03] hashar: just iphone! :) but still, it fixed iphone :) [14:45:07] it did not break anything [14:45:54] zeljkof: hashar: yeah that looks great! thanks so much :) [14:46:17] AndyRussG: crazy hacks going on :) [14:46:42] AndyRussG: we also found out the browser tests suffers from some lame initialization that makes unecessarly slow [14:46:48] heheh sounds interesting at least ;p [14:47:00] hmm [14:47:53] Yeah I did notice they take a while... [14:48:13] 10Continuous-Integration, 6operations: Build Debian package jenkins-debian-glue for Jessie - https://phabricator.wikimedia.org/T95006#1177941 (10hashar) I have manually installed the binary packages I needed on the integration-slave-jessie-1001.eqiad.wmflabs instance. ``` dpkg -i jenkins-debian-glue_0.11.0_al... [14:49:00] !log integration-slave-jessie-1001 : manually installed jenkins-debian-glue Debian packages. It is pending upload by ops to apt.wikimedia.org {{bug|T95006}} [14:49:04] Logged the message, Master [14:51:01] Yippee, build fixed! [14:51:01] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #567: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/567/ [15:10:43] Yippee, build fixed! [15:10:43] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hi,contintLabsSlave && UbuntuTrusty build #36: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hi,label=contintLabsSlave%20&&%20UbuntuTrusty/36/ [16:38:11] PROBLEM - Citoid on deployment-sca01 is CRITICAL: Connection refused [17:29:28] 10Continuous-Integration, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1178354 (10faidon) Preliminary analysis: - The libvips/libmemcached are puppet bugs. These are libraries with a SONAME suffix and we shouldn't hardcode their SONAM... [18:00:13] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #497: FAILURE in 2 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/497/ [18:02:18] Is integration-slave1401 a new-ish instance? [18:02:40] Aaron is hitting this error there -- https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/10575/consoleFull [18:02:42] 10Deployment-Systems, 7Documentation: document trebuchet - https://phabricator.wikimedia.org/T94619#1178530 (10thcipriani) a:3thcipriani [18:03:15] Which looks to me to be a ulimit problem of some sort -- "[Fri Apr 3 17:51:33 2015] [hphp] [5195:7f4149b3fbc0:0:000001] [948484:9cebc9:9d070f:a6e441:856a5d:7f41417bbec5:909e04] Unable to set CoreFileSize to 8589934592: Operation not permitted (1)" [18:03:37] march 6: 21:36 Krinkle: Provisioning integration-slave1401 - integration-slave1404 [18:03:59] hmm around for a month [18:04:14] One would think we'd have noticed if all hhvm runs failed there [18:05:31] argh. I can't see https://integration.wikimedia.org/ci/computer/integration-slave1401/builds to see if there is a trend [18:06:24] me neither [18:07:51] Eh. The php5 build has the same error without the corefile possible cause -- https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/10543/console [18:08:02] Looks like its something else [18:08:17] * bd808 will report back to Aaron and then eat lunch [18:08:21] 10Continuous-Integration, 6Release-Engineering, 7Browser-Tests, 5Patch-For-Review: It takes about 20 seconds just to start a Sauce Labs browser - https://phabricator.wikimedia.org/T92613#1178542 (10dduvall) >>! In T92613#1167682, @dduvall wrote: > https://github.com/watir/watir-webdriver/blob/97b54739dedbd... [18:09:10] bd808: I'll report a task about the lack of build history view, though [18:10:09] *nod* it's handy on occasion [18:10:32] 10Continuous-Integration: Can't view per-slave build history - https://phabricator.wikimedia.org/T95024#1178543 (10greg) 3NEW [18:13:28] 10Continuous-Integration: Can't view per-slave build history - https://phabricator.wikimedia.org/T95024#1178554 (10greg) Well, looks like it is due to Jenkins never being able to figure out how to parsing long files efficiently: {rOPUPe8a9f5be5a5ce7a04e3894c5ad3df5802d9c7559} {T63964} Antoine's last comment o... [18:14:14] 10Continuous-Integration: Can't view per-slave build history - https://phabricator.wikimedia.org/T95024#1178561 (10greg) [18:32:11] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #274: FAILURE in 2 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/274/ [18:55:09] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #273: FAILURE in 2 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/273/ [18:59:29] 10Continuous-Integration: Can't view per-slave build history - https://phabricator.wikimedia.org/T95024#1178794 (10hashar) 5Open>3declined a:3hashar @legoktm and @krinkle worked hard in March to reduce the number of jobs; we still have 2300 or so and a good chunk of them are build on labs slaves so that wo... [19:13:59] twentyafterfour, hi [19:14:11] hi [19:15:10] twentyafterfour, can you reply to https://phabricator.wikimedia.org/T93543#1172577 please? [19:15:25] (but like, on the task, not in this public channel, please) [19:16:57] Krenair: reading... [19:34:50] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1729: FAILURE in 48 min: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1729/ [19:51:12] 6Release-Engineering: Describe input and actions that trigger work by Release Engineering people - https://phabricator.wikimedia.org/T89720#1178892 (10greg) 5Open>3declined a:3greg Decline for now until the team offsite. [19:51:13] 6Release-Engineering, 3releng-201415-Q3: [Quarterly Success Metric] RelEng+TPG process discussion and improvements (tracking) - https://phabricator.wikimedia.org/T88708#1178895 (10greg) [19:52:25] 10Continuous-Integration, 6Release-Engineering: Amir does not have +2 on integration/config - https://phabricator.wikimedia.org/T86629#1178900 (10greg) @Amire80: thoughts on Timo's comment? [19:53:29] 6Release-Engineering: Describe process for setting, reporting, and addressing priority of Release Engineering work - https://phabricator.wikimedia.org/T89722#1178901 (10greg) 5Open>3declined a:3greg Decline for now until the team offsite. [19:53:30] 6Release-Engineering, 3releng-201415-Q3: [Quarterly Success Metric] RelEng+TPG process discussion and improvements (tracking) - https://phabricator.wikimedia.org/T88708#1178904 (10greg) [19:55:42] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 7Documentation: Document how to tag extensions in git - https://phabricator.wikimedia.org/T94412#1178907 (10greg) @mmodell has some thoughts on this I presume. [19:56:45] (03PS1) 10BryanDavis: Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 [19:57:03] (03CR) 10jenkins-bot: [V: 04-1] Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [19:59:22] (03PS1) 10Dduvall: Renamed `bind` methods to `configure` [selenium] - 10https://gerrit.wikimedia.org/r/201830 [20:11:31] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 7Documentation: Document how to tag extensions in git - https://phabricator.wikimedia.org/T94412#1178953 (10mmodell) I wholeheartedly support the tagging of releases. I think everyone should have permission to push tags to the same repos they have the ability... [20:16:51] (03PS2) 10BryanDavis: Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 [20:20:33] (03CR) 10BryanDavis: "This patch is silly to the point of being ridiculous, but I've had it laying around in my git repo for over a year and I decided that I wo" [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:20:51] bd808: I love you [20:21:12] ascii art makes life worth living [20:31:56] (03CR) 10Yuvipanda: [C: 031] "<3" [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:34:07] (03CR) 10Greg Grossmeier: [C: 031] Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:38:49] marktraceur: https://phabricator.wikimedia.org/T94915 [20:38:52] you aren't dan [20:38:59] dan's sitting right there but not online [20:39:24] <^d> thcipriani: Did you disable puppet on staging-palladium? [20:40:01] ^d: not that I remember, but I noticed that the other evening and forgot the mention it :\ [20:40:12] <^d> YuviPanda: ? [20:40:27] not me [20:40:32] <^d> Hmmm [20:40:51] actually not sure how to re-enable it. But I don't think there's any harm in doing so. [20:41:48] oh: Use 'puppet agent --enable' to re-enable. [20:44:51] 10Continuous-Integration, 6operations, 5Patch-For-Review, 7Puppet, 7Regression: Puppet: "Package[git-core] is already declared in file modules/authdns/manifests/scripts.pp" - https://phabricator.wikimedia.org/T94921#1179104 (10Dzahn) should be fixed now. wanna confirm? [20:47:38] (03CR) 10Thcipriani: [C: 031] "http://tyler.zone/scap.png ← in case anyone was curious" [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:48:04] thcipriani: slick domain [20:48:05] (03CR) 10Ori.livneh: [C: 031] "Omfg finally yes." [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:48:28] thcipriani: where do you register .zone? [20:48:33] YuviPanda: it's one of the few "tyler" gtlds I could grab that wasn't like tyler.limo :) [20:48:45] YuviPanda: just on namecheap [20:49:17] thcipriani: damn, danger.zone is not available [20:50:18] fwiw, I also own http://gangsta.party [20:50:37] wow [20:50:40] thcipriani: I have usedtobe.sexy [20:50:50] heh, solid. [20:51:18] (03CR) 10Dzahn: [C: 031] Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:54:16] alright, who's going to take the honor of merging? [20:55:09] 10Continuous-Integration, 7Upstream: Zuul Status API cached too long by Varnish - https://phabricator.wikimedia.org/T94796#1179140 (10Krinkle) p:5Triage>3Normal a:3Krinkle [20:55:23] 10Continuous-Integration, 7Upstream: Zuul Status API cached too long by Varnish - https://phabricator.wikimedia.org/T94796#1172819 (10Krinkle) https://review.openstack.org/#/c/170081/ [20:57:41] (03CR) 1020after4: [C: 032] "I'll merge this when pigs fly." [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [20:58:02] (03Merged) 10jenkins-bot: Add a logo banner to scap [tools/scap] - 10https://gerrit.wikimedia.org/r/201829 (owner: 10BryanDavis) [21:03:53] thcipriani: btw, I vaguely saw you do a mention about unifying parsoid puppet roles. see also https://gerrit.wikimedia.org/r/#/c/193082/ [21:04:11] thcipriani: it mostly works, I just haven't had time to 1. JFDI 2. verify it still works with jenkins 3. check if it works with jenkins atm at all [21:04:38] <^d> Do you know it's impossible to add a first wiki? [21:04:48] <^d> not by hand, that is [21:05:08] !log Delete unfinished/unpoooled instances integration-slave-precise-1011-1014. (T94916) [21:05:14] Logged the message, Master [21:07:01] PROBLEM - Host integration-slave-precise-1012 is DOWN: CRITICAL - Host Unreachable (10.68.17.209) [21:07:01] YuviPanda: mine was https://gerrit.wikimedia.org/r/#/c/201636/ seems like there are a few hard beta requirements that should stick around. Otherwise I paramaterized all the differences. Would be nice to find a spot to move the remainder of the beta role :\ [21:07:49] thcipriani: what are the differences? [21:08:04] thcipriani: everything should use upstart now (and does!) and the jenkins stuff has its own role now (I mentioned this in comments on my patch) [21:08:05] PROBLEM - Host integration-slave-precise-1011 is DOWN: CRITICAL - Host Unreachable (10.68.17.130) [21:08:45] PROBLEM - Host integration-slave-precise-1013 is DOWN: CRITICAL - Host Unreachable (10.68.17.239) [21:09:16] YuviPanda: so without the upstart thing, it's mostly the role::ci::slave::labs::common which could be included on an instance elsewhere and contint::slave-scripts [21:09:41] 6Release-Engineering, 6Phabricator: Adding users to CC on Phabricator security tasks doesn't add them to the view/edit policy - https://phabricator.wikimedia.org/T94565#1179222 (10mmodell) @csteipp: I wrote a custom policy rule class called 'subscribers of task' which should be added instead of individual use... [21:09:41] thcipriani: yup, I made a role with just those, and already mathoid, cxserver, etc are using that separately [21:09:45] oh, and the jenkins-deploy user sudo permission for restarting [21:09:46] PROBLEM - Host integration-slave-precise-1014 is DOWN: CRITICAL - Host Unreachable (10.68.18.2) [21:09:55] thcipriani: that's on wikitech, sudo policy special page :D [21:10:26] beautiful, then should be able to ditch beta entirely. [21:10:33] thcipriani: yup! That's what my patch does :D [21:10:43] thcipriani: thtere's even a beta instance where it works fine [21:11:01] <^d> !log puppet re-enabled on staging-palladium, running fine again [21:11:03] Logged the message, Master [21:11:12] thcipriani: it just needed some time to push it through, but I had basically been distracted, and hashar -1'd it and then I had to travel, etc etc [21:11:25] thcipriani: I'm ok with JFDI now if you're ok babysitting it through deployment-prep [21:11:33] probably not now, though [21:11:37] friday evening >_>: [21:11:59] YuviPanda: yup, whenever you're ready to deploy it, I'll watch it :) [21:12:09] thcipriani: what TZ are you in? [21:12:15] Mountain [21:12:19] thcipriani: should we just do it? I don't think there should be any effects on prod... [21:13:11] YuviPanda: should you change manifests/site.pp from ::common to nothing? [21:13:21] did I not do that? [21:13:22] * YuviPanda see [21:13:23] s [21:13:48] <^d> Hmm, I'm trying to figure out the best way to tidy up mw-config in a way that doesn't require duplication for beta/staging [21:14:15] ^d: ultimately BeCaaS, so it should be generalizble to N environments.. [21:15:36] <^d> Yeah I'd like something that's clean [21:15:45] <^d> And not just if( foo ) {} else {} [21:16:48] 10Deployment-Systems, 7Documentation: Document Scap - https://phabricator.wikimedia.org/T94618#1179245 (10mmodell) 5Open>3Resolved I updated wikitech's entry on scap to include the diagram and a link to the docs on doc.mediawiki.org. [21:17:49] 10Deployment-Systems, 7Epic: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1179249 (10mmodell) @aklapper: yeah it's epic [21:18:37] totally epic dude [21:23:52] YuviPanda: so all the other *oid roles have a ::jenkins-access, but parsoid doesn't have one of those. Shouldn't it? [21:24:27] thcipriani: no, all the others should be killed. they have the same code... [21:24:44] thcipriani: I made a CI role for it, don't remember what it's called but it's there in the comments section [21:24:52] also is hiera('has_lvs', true) a thing that we're sticking with? [21:25:17] thcipriani: looks like, it's in other places too [21:26:02] YuviPanda: then 193082 should wrap the lvs::realserver up [21:26:51] thcipriani: hmm, true [21:27:02] thcipriani: only negative effect it has is spamming puppetlog but let me do it anyway [21:29:10] and I see now the role::ci::jenkins_access now. Not having taken a deep dive in to anything except for parsoid it lgtm :) [21:31:42] YuviPanda: That being said, let's not make this a late in the day Friday labs deploy. I just saw our old SparkFun deploy flow chart, I think it's good policy: https://www.flickr.com/photos/sparkfun/14731247948/ [21:31:55] thcipriani: yeah, totally :) [21:31:57] !log Upgrading npm from v2.4.1 to v2.7.5 (rolling, slave by slave graceful) [21:32:00] Logged the message, Master [21:32:00] thcipriani: let's do this on Monday [21:32:10] YuviPanda: sounds good. [21:50:48] 10Continuous-Integration, 5Patch-For-Review: Fix npm oid jobs - https://phabricator.wikimedia.org/T92369#1179327 (10greg) p:5Triage>3Normal [21:57:40] 6Release-Engineering, 6Phabricator: Adding users to CC on Phabricator security tasks doesn't add them to the view/edit policy - https://phabricator.wikimedia.org/T94565#1179342 (10csteipp) >>! In T94565#1179222, @mmodell wrote: > @csteipp: I wrote a custom policy rule class called 'subscribers of task' which... [22:03:02] 6Release-Engineering, 6Phabricator: Adding users to CC on Phabricator security tasks doesn't add them to the view/edit policy - https://phabricator.wikimedia.org/T94565#1179368 (10csteipp) In testing, this looks like it's working. What can I do the next time this comes up to help you debug? [22:48:39] 10Deployment-Systems, 7Epic: EPIC: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1179463 (10mmodell) [22:55:18] !log Updated scap to a1a5235 (Add a logo banner to scap) [22:55:21] Logged the message, Master [22:58:37] 6Release-Engineering, 6Phabricator: Adding users to CC on Phabricator security tasks doesn't add them to the view/edit policy - https://phabricator.wikimedia.org/T94565#1179475 (10mmodell) @csteipp: right, it should just work™ If it comes up again just link me directly to the issue before changing anything so... [23:01:10] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce build #9: FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce/9/ [23:08:46] !log Finished npm upgrade on precise slaves. Rolling on trusty slaves now. [23:08:48] Logged the message, Master [23:20:54] (03PS2) 10Krinkle: Publish oojs code coverage to integration.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) [23:27:52] 10Continuous-Integration, 7Upstream: Fails npm build failure "File exists: ../esprima/bin/esparse.js" - https://phabricator.wikimedia.org/T90816#1179547 (10Krinkle) 5Open>3Resolved a:3Krinkle I've upgraded npm from v2.4.1 to v2.7.6 on our slaves. I've worked with upstream and there've been several improv... [23:31:34] !log Finished npm upgrade on trusty slaves. [23:47:24] !log for Krinkle 23:31 "Finished npm upgrade on trusty slaves." [23:47:26] Logged the message, Master [23:47:35] Thx [23:51:33] (03PS3) 10Krinkle: Publish oojs code coverage to integration.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) [23:58:10] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce build #9: FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce/9/