[00:36:45] (03PS5) 10Awight: DonationInterface should track master in the main cluster [tools/release] - 10https://gerrit.wikimedia.org/r/230705 [00:55:38] PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [02:10:14] (03PS2) 10Thcipriani: Add config deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) [02:11:51] (03CR) 10Krinkle: [C: 031] DonationInterface should track master in the main cluster [tools/release] - 10https://gerrit.wikimedia.org/r/230705 (owner: 10Awight) [04:17:23] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n, 7Wikimedia-log-errors: l10n-update not updating Vector and extensions - https://phabricator.wikimedia.org/T103879#1664960 (10mmodell) If the problem was just localizationupdate, then today's full scap would have resolved the issue. Tha... [05:20:19] !log beta-scap-eqiad and others stuck in Jenkins queue for over 6 hours. deployment-bastion.eqiad is idle but not accepting jobs. [05:20:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [05:24:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #201: ABORTED in 8 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/201/ [05:24:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #550: ABORTED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/550/ [05:26:10] !log Graceful restarting of Jenkins [05:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [05:32:14] !log beta-scap-eqiad is back up [05:32:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [05:45:22] (03CR) 1020after4: [C: 031] Add config deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) (owner: 10Thcipriani) [05:46:05] (03CR) 1020after4: [C: 031] Simplify logging in ssh module [tools/scap] - 10https://gerrit.wikimedia.org/r/238959 (owner: 10Chad) [06:09:29] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 5Patch-For-Review, and 2 others: "Nonce already used" regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1665186 (10jayvdb) p:5High>3Unbreak! [06:37:20] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:08:31] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n, 7Wikimedia-log-errors: l10n-update not updating Vector and extensions - https://phabricator.wikimedia.org/T103879#1665309 (10KartikMistry) Also see: T112964 [07:24:47] 10Deployment-Systems, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, and 4 others: is shown instead of the actual message on Special:ContentTranslation in ... - https://phabricator.wikimedia.org/T112964#1665343 [07:41:48] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Improve scap's output - https://phabricator.wikimedia.org/T109858#1665394 (10mmodell) interactive scap, controlling tmux, auto-completing scap built-in commands (e.g. cli.Application sub-classes) and also handling threaded / async subprocess i/o: {F2627608} [07:42:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #169: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/169/ [08:10:47] zeljkof-meeting: wanna try firefox one ? [08:51:57] 6Release-Engineering-Team, 7Epic, 5Patch-For-Review, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1665523 (10zeljkofilipin) a:5dduvall>3zeljkofilipin [08:58:05] zeljkof-meeting: https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/selenium,n,z :D [09:04:58] zeljkof-meeting: in Gerrit I use https://gerrit.wikimedia.org/r/#/q/is:open+reviewer:self+label:code-review%253D0%252Cself,n,z [09:05:25] andre__: does Gerrit clean up day has any specific IRC channel ? [09:06:09] hashar, just generic #wikimedia-dev if someone wants to coordinate [09:06:21] but not sure how much interaction there will be [09:06:44] andre__: ok ok :-} Will clean up the integration/config repo. Will probably keep me busy all day long anyway [09:07:03] Yay, cool. Thanks a lot! [09:07:08] hashar: firefox hello was actually and improvement over hangouts [09:07:22] especially since hangouts did not want to cooperate today [09:09:49] andre__: and I found out that https://gerrit.wikimedia.org/r/#/q/is:open+reviewer:self+label:code-review%253D0%252Cself,n,z is very nice to fetch all patches one has to review [09:11:38] (03CR) 10Hashar: [C: 031] Simplify logging in ssh module [tools/scap] - 10https://gerrit.wikimedia.org/r/238959 (owner: 10Chad) [09:16:32] (03Abandoned) 10Hashar: Remove experimental support for php composer test in UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239870 (owner: 10Paladox) [09:20:37] (03PS2) 10Hashar: Move jshint test to check: in UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:20:44] (03PS3) 10Hashar: Move jshint test to check: in UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:21:50] (03CR) 10Hashar: [C: 032] Move jshint test to check: in UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:22:46] (03Merged) 10jenkins-bot: Move jshint test to check: in UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:22:54] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:24:08] GRRR [09:24:52] (03CR) 10Hashar: "Paladox: no need to Thanks on commits being merged, although it is appreciated. The reason is sends us unnecessary notification emails :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/239869 (owner: 10Paladox) [09:25:48] (03PS1) 10Paladox: [UniversalLanguageSelector] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240324 [09:26:37] (03CR) 10Paladox: "This https://gerrit.wikimedia.org/r/#/c/239872/ can be merged after this is merged." [integration/config] - 10https://gerrit.wikimedia.org/r/240324 (owner: 10Paladox) [09:29:09] (03Abandoned) 10Hashar: [UniversalLanguageSelector] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240324 (owner: 10Paladox) [09:29:17] (03PS3) 10Hashar: Add composer-test to UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239873 (owner: 10Paladox) [09:31:24] Yippee, build fixed! [09:31:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #730: FIXED in 1 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/730/ [09:32:42] (03PS4) 10Hashar: Add composer-test to UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239873 (owner: 10Paladox) [09:33:10] (03CR) 10Hashar: [C: 032] Add composer-test to UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239873 (owner: 10Paladox) [09:34:01] (03Merged) 10jenkins-bot: Add composer-test to UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/239873 (owner: 10Paladox) [09:34:40] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/239873 (owner: 10Paladox) [09:36:31] (03PS1) 10Paladox: [UniversalLanguageSelector] Remove experimental: [integration/config] - 10https://gerrit.wikimedia.org/r/240325 [09:36:41] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1665724 (10phuedx) From my email reply to the latest nightly build failures: I merged the test survey con... [09:38:17] (03Abandoned) 10Paladox: Add composer-test to UniversalLanguageSelector [integration/config] - 10https://gerrit.wikimedia.org/r/228880 (owner: 10Paladox) [09:46:13] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1665732 (10hashar) Recreating integration-slave-trusty-1014 and int... [09:47:08] PROBLEM - Host integration-slave-trusty-1021 is DOWN: CRITICAL - Host Unreachable (10.68.16.8) [09:47:40] (03CR) 10Hashar: [C: 04-1] "Need to complete the PHPCS integration with composer at : https://gerrit.wikimedia.org/r/#/c/228981/" [integration/config] - 10https://gerrit.wikimedia.org/r/239875 (owner: 10Paladox) [09:48:23] (03PS1) 10Paladox: [Translate] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240326 [09:50:03] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1628865 (10phuedx) I've moved this into **-1 (Needs More Work)** per T112204#1665724. [09:52:46] (03CR) 10Paladox: "jshint passes please see https://integration.wikimedia.org/ci/job/jshint/3499/console so we can convert to npm." [integration/config] - 10https://gerrit.wikimedia.org/r/228872 (owner: 10Paladox) [09:52:49] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 needs to be deployed on RESTBase boxes and needs a group on tin - https://phabricator.wikimedia.org/T109862#1665747 (10hashar) Still pending outcome of discussion/review on https://gerrit.wikimedia.org/r/#/c/232843/ [09:57:31] Yippee, build fixed! [09:57:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #567: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/567/ [10:01:25] (03PS4) 10Hashar: Add composer-test to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229180 (owner: 10Paladox) [10:01:59] (03CR) 10Hashar: [C: 032] "Entry point is added by https://gerrit.wikimedia.org/r/#/c/229174/" [integration/config] - 10https://gerrit.wikimedia.org/r/229180 (owner: 10Paladox) [10:02:50] (03Merged) 10jenkins-bot: Add composer-test to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229180 (owner: 10Paladox) [10:02:54] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/229180 (owner: 10Paladox) [10:05:59] (03PS1) 10Paladox: [Slate] Remove mw-check-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240331 [10:06:57] (03CR) 10Paladox: "@Hashar would the php lint test still run for non-whitelisted users or would we need to add the php lint test." [integration/config] - 10https://gerrit.wikimedia.org/r/240331 (owner: 10Paladox) [10:12:06] (03CR) 10Hashar: [C: 04-1] "Gotta finish up the Translate entry point first: https://gerrit.wikimedia.org/r/#/c/240327/" [integration/config] - 10https://gerrit.wikimedia.org/r/240326 (owner: 10Paladox) [10:14:01] (03CR) 10Hashar: [C: 04-1] "Need to complete the npm entry point first: https://gerrit.wikimedia.org/r/#/c/228864/" [integration/config] - 10https://gerrit.wikimedia.org/r/228872 (owner: 10Paladox) [10:17:31] (03PS2) 10Hashar: [UniversalLanguageSelector] Remove experimental: [integration/config] - 10https://gerrit.wikimedia.org/r/240325 (owner: 10Paladox) [10:17:49] (03CR) 10Hashar: [C: 032] [UniversalLanguageSelector] Remove experimental: [integration/config] - 10https://gerrit.wikimedia.org/r/240325 (owner: 10Paladox) [10:18:41] (03Merged) 10jenkins-bot: [UniversalLanguageSelector] Remove experimental: [integration/config] - 10https://gerrit.wikimedia.org/r/240325 (owner: 10Paladox) [10:19:14] (03Abandoned) 10Hashar: Update translate tests, Remove old phpcs test [integration/config] - 10https://gerrit.wikimedia.org/r/228871 (owner: 10Paladox) [10:33:06] (03CR) 10Hashar: [C: 04-1] "Need to complete the entry point https://gerrit.wikimedia.org/r/#/c/228273/" [integration/config] - 10https://gerrit.wikimedia.org/r/228277 (owner: 10Paladox) [10:34:23] cleaning up cleaning up :D [10:34:24] https://gerrit.wikimedia.org/r/#/q/is:open+reviewer:self+label:code-review%253D0%252Cself+NOT+label:code-review%253D-1+NOT+label:code-review%253D-2+NOT+label:code-review%253D2+NOT+label:verified%253D-1,n,003779dc00036f0a [10:34:43] actually https://gerrit.wikimedia.org/r/#/q/is:open+reviewer:self+label:code-review%253D0%252Cself+NOT+label:code-review%253D-1+NOT+label:code-review%253D-2+NOT+label:code-review%253D2+NOT+label:verified%253D-1,n,z [10:44:14] (03PS8) 10Hashar: Update ShoutWikiAds tests [integration/config] - 10https://gerrit.wikimedia.org/r/225298 (owner: 10Paladox) [10:46:28] (03CR) 10Hashar: [C: 032] "Deleting Jenkins job mwext-ShoutWikiAds-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/225298 (owner: 10Paladox) [10:47:13] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/225298 (owner: 10Paladox) [10:47:38] Paladox doesn't listen :( [10:48:10] (03Merged) 10jenkins-bot: Update ShoutWikiAds tests [integration/config] - 10https://gerrit.wikimedia.org/r/225298 (owner: 10Paladox) [11:02:00] !log restarted integration puppetmaster. Newly created instances fail with: [1;31mError: Could not request certificate: Connection refused - connect(2)[0m [11:02:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:09:14] PROBLEM - Host integration-slave-trusty-1017 is DOWN: CRITICAL - Host Unreachable (10.68.20.91) [11:09:41] PROBLEM - Host integration-slave-trusty-1014 is DOWN: CRITICAL - Host Unreachable (10.68.20.90) [11:09:49] ^^ I have deleted them [11:11:04] 6Release-Engineering-Team, 7Epic, 5Patch-For-Review, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week - https://phabricator.wikimedia.org/T94150#1665858 (10zeljkofilipin) [11:15:25] 10Browser-Tests: Fix or delete browsertests* Jenkins jobs that are failing for more than a week - https://phabricator.wikimedia.org/T113463#1665868 (10zeljkofilipin) 3NEW a:3zeljkofilipin [11:16:27] !log applying contint::slave::labs to integration-slave-trusty-1014 and integration-slave-trusty-1017 [11:16:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:20:32] (03Abandoned) 10Hashar: Update BreadCrumbs tests [integration/config] - 10https://gerrit.wikimedia.org/r/228136 (owner: 10Paladox) [11:20:52] (03PS13) 10Hashar: Update BreadCrumbs tests [integration/config] - 10https://gerrit.wikimedia.org/r/225291 (owner: 10Paladox) [11:21:39] (03CR) 10Hashar: [C: 032] "Entry point added by https://gerrit.wikimedia.org/r/#/c/225690/" [integration/config] - 10https://gerrit.wikimedia.org/r/225291 (owner: 10Paladox) [11:22:29] (03Merged) 10jenkins-bot: Update BreadCrumbs tests [integration/config] - 10https://gerrit.wikimedia.org/r/225291 (owner: 10Paladox) [11:24:41] (03PS2) 10Hashar: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [11:24:47] (03PS3) 10Hashar: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [11:28:30] (03CR) 10Hashar: [C: 04-1] "npm does not work on the source repository :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [11:31:23] (03CR) 10Hashar: [C: 04-1] "Need the initial commit to add composer/npm entry points: https://gerrit.wikimedia.org/r/#/c/229423/" [integration/config] - 10https://gerrit.wikimedia.org/r/239535 (owner: 10Paladox) [11:32:43] (03PS4) 10Hashar: Add jenkins tests for ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/239671 (https://phabricator.wikimedia.org/T113049) (owner: 10Paladox) [11:32:52] (03CR) 10Hashar: [C: 032] Add jenkins tests for ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/239671 (https://phabricator.wikimedia.org/T113049) (owner: 10Paladox) [11:33:27] (03Abandoned) 10Sbisson: [WIP] Support pages with underscore like User_talk [selenium] - 10https://gerrit.wikimedia.org/r/224809 (owner: 10Sbisson) [11:33:41] (03Merged) 10jenkins-bot: Add jenkins tests for ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/239671 (https://phabricator.wikimedia.org/T113049) (owner: 10Paladox) [11:34:43] RECOVERY - Host integration-slave-trusty-1014 is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms [11:34:51] RECOVERY - Host integration-slave-trusty-1017 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [11:34:53] (03CR) 10Hashar: "Changes currently opened on ArticlePlaceholder are passing just fine." [integration/config] - 10https://gerrit.wikimedia.org/r/239671 (https://phabricator.wikimedia.org/T113049) (owner: 10Paladox) [11:35:37] PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:35:58] food time [11:40:07] PROBLEM - Puppet failure on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:45:40] RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:06] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [12:31:57] !log pooling integration-slave-trusty-1014 and integration-slave-trusty-1017 in Jenkins [12:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:33:34] !sal [12:33:34] https://tools.wmflabs.org/sal/releng [12:35:13] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1666142 (10hashar) 5Open>3Resolved We have rebuild: * integrati... [12:35:22] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1666144 (10hashar) (npm / grunt-cli are up-to-date) [12:35:48] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests, 5Patch-For-Review: localStorage qunit tests causing failures with Chrome 45 - https://phabricator.wikimedia.org/T113413#1666145 (10Jdforrester-WMF) 5Open>3Resolved [12:41:31] (03PS7) 10Hashar: Update Apex tests [integration/config] - 10https://gerrit.wikimedia.org/r/226994 (owner: 10Paladox) [12:43:24] (03CR) 10Hashar: [C: 032] Update Apex tests [integration/config] - 10https://gerrit.wikimedia.org/r/226994 (owner: 10Paladox) [12:43:46] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/226994 (owner: 10Paladox) [12:44:14] (03Merged) 10jenkins-bot: Update Apex tests [integration/config] - 10https://gerrit.wikimedia.org/r/226994 (owner: 10Paladox) [12:53:53] When I click "Rebase Change" in Gerrit on an ancient patchset, and I get "The change could not be rebased due to a path conflict during merge", am I supposed to CR-1 ? Or what's recommended? [12:55:58] andre__: Yeah, you can CR-1 and comment it needs manual rebase [12:56:10] (03PS1) 10Hashar: mw/ext/ProxyListDb has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/240347 [12:56:18] Reedy, thanks [12:56:25] (03CR) 10Hashar: [C: 032] mw/ext/ProxyListDb has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/240347 (owner: 10Hashar) [12:57:15] (03Merged) 10jenkins-bot: mw/ext/ProxyListDb has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/240347 (owner: 10Hashar) [12:58:47] (03CR) 10Paladox: "Please see https://gerrit.wikimedia.org/r/#/c/240349/" [integration/config] - 10https://gerrit.wikimedia.org/r/239535 (owner: 10Paladox) [13:03:57] (03Abandoned) 10Hashar: Add composer-test to SpamDiffTool [integration/config] - 10https://gerrit.wikimedia.org/r/236210 (owner: 10Paladox) [13:05:31] (03PS3) 10Hashar: composer/npm jobs for SpamDiffTool [integration/config] - 10https://gerrit.wikimedia.org/r/236203 (owner: 10Paladox) [13:06:23] (03CR) 10Hashar: [C: 04-1] "Now provides both composer and npm triggers." [integration/config] - 10https://gerrit.wikimedia.org/r/236203 (owner: 10Paladox) [13:08:53] (03CR) 10Paladox: "Fixed it in https://gerrit.wikimedia.org/r/#/c/240353/ please re try." [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [13:10:15] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666256 (10hashar) 3NEW a:3hashar [13:13:36] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666265 (10hashar) The instance is m1.small which has 20GB disk. Most of it being allocated to / : ``` # df -h Filesystem Size Used Avail Use% M... [13:14:31] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666266 (10hashar) I have deleted all content from `/mnt/jenkins-workspace/workspace` and reconnected the instance to unblock jobs. [13:16:35] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666267 (10hashar) Creating integration-lightslave-jessie-1003.integration.eqiad.wmflabs as ci1.medium (2CPU, 40GB disk, 2GB RAM). [13:17:38] (03PS4) 10Paladox: Add jshint to check: to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229178 [13:20:08] (03CR) 10Paladox: "Ok yes somebody please merge this." [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [13:21:52] (03CR) 10JanZerebecki: "Thx." [integration/config] - 10https://gerrit.wikimedia.org/r/239671 (https://phabricator.wikimedia.org/T113049) (owner: 10Paladox) [13:22:07] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1666269 (10JanZerebecki) Will this extension depend on Wikibase? Certainly on Scribunto. [13:22:19] (03CR) 10Paladox: Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [13:22:43] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1666270 (10JanZerebecki) Also missing are probably jobs for: npm, composer, qunit [13:24:41] (03CR) 10Paladox: Add composer-test to interwiki [integration/config] - 10https://gerrit.wikimedia.org/r/225313 (owner: 10Paladox) [13:24:45] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666271 (10hashar) p:5Unbreak!>3Normal [13:25:04] (03CR) 10Paladox: Update ConfirmAccount tests [integration/config] - 10https://gerrit.wikimedia.org/r/225311 (owner: 10Paladox) [13:25:10] 10Continuous-Integration-Infrastructure: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#1666256 (10hashar) Cant create Jessie image on labs for now. They are stalled trying to contact the LDAP server or something. [13:31:18] (03CR) 10Hashar: [C: 04-1] "Waiting for https://gerrit.wikimedia.org/r/#/c/236067" [integration/config] - 10https://gerrit.wikimedia.org/r/236068 (owner: 10Paladox) [13:36:33] (03CR) 10Hashar: [C: 04-1] Change jslint test to jshint test in CheckUser (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/225182 (owner: 10Paladox) [13:37:24] (03CR) 10Hashar: "Should get the composer job as well from https://gerrit.wikimedia.org/r/#/c/234856/" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/225182 (owner: 10Paladox) [13:37:40] (03Abandoned) 10Hashar: Add composer-test test to CheckUser [integration/config] - 10https://gerrit.wikimedia.org/r/234856 (owner: 10Paladox) [13:41:28] (03CR) 10Hashar: [C: 04-1] "The source repository patch is not ready yet https://gerrit.wikimedia.org/r/#/c/225034/" [integration/config] - 10https://gerrit.wikimedia.org/r/225035 (owner: 10Paladox) [13:43:12] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1666306 (10zeljkofilipin) I am not sure, but looks like the problem is not that the header changed: ``` $ git log --grep X-CS commit 0d44661ad18e3030342fa475f94014635e4069a6 Autho... [13:46:13] PROBLEM - Host mira is DOWN: CRITICAL - Host Unreachable (10.68.17.215) [13:47:49] (03PS8) 10Hashar: Update EditSimilar tests [integration/config] - 10https://gerrit.wikimedia.org/r/228641 (owner: 10Paladox) [13:47:57] (03CR) 10Hashar: [C: 032] Update EditSimilar tests [integration/config] - 10https://gerrit.wikimedia.org/r/228641 (owner: 10Paladox) [13:48:12] (03CR) 10jenkins-bot: [V: 04-1] Update EditSimilar tests [integration/config] - 10https://gerrit.wikimedia.org/r/228641 (owner: 10Paladox) [13:48:29] (03Merged) 10jenkins-bot: Update EditSimilar tests [integration/config] - 10https://gerrit.wikimedia.org/r/228641 (owner: 10Paladox) [13:49:41] (03CR) 10Hashar: "Seems to work fine according to https://gerrit.wikimedia.org/r/#/c/240360/" [integration/config] - 10https://gerrit.wikimedia.org/r/228641 (owner: 10Paladox) [13:50:06] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1666323 (10zeljkofilipin) Looks like mediawiki receives the headers. Look for lines with `X-CS`. ``` vagrant@mediawiki-vagrant:/vagrant/logs$ grep X-CS *.log -C 5 mediawiki-wiki-de... [13:52:11] !log dist upgrading integration-slave-trusty1014 and integration-slave-trusty-1017 to make sure we get the latest versions from apt.wm.org [13:52:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:09:34] 10Beta-Cluster, 10pywikibot-core, 5Patch-For-Review: Link.langlinkUnsafe does not work on Beta-Cluster wikis - https://phabricator.wikimedia.org/T112006#1666356 (10demon) 5Open>3Resolved [14:12:08] 10Beta-Cluster, 10pywikibot-core, 5Patch-For-Review: Link.langlinkUnsafe does not work on Beta-Cluster wikis - https://phabricator.wikimedia.org/T112006#1666362 (10XZise) How long does it take until that comes live so that we can verify that it worked? At the moment the English Wikipedia still advertises `ba... [14:22:11] thcipriani: ldaplist -l passwd mwdeploy .. Ie the user is defined in labs ldap [14:23:30] thcipriani: https://wikitech.wikimedia.org/wiki/Special:NovaSudoer gotcha #2 [14:30:51] thcipriani: not so secret repo is labs/private.git [14:35:34] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #269: FAILURE in 7 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/269/ [14:35:38] Hmmm. [14:35:46] Krenair: T112006 [14:36:48] thanks, will check in a sec [14:44:45] Krenair: Also, https://gerrit.wikimedia.org/r/#/c/236045/. "Do not merge until Sept. 15" [14:44:47] Is that ok now? [14:45:21] well it is the 23rd of september [14:45:31] Heh, it is! [14:45:31] but I'd ask greg-g [14:47:47] * ostriches just read the full task [14:47:52] You know, I'm not gonna touch that one! [14:47:53] :) [14:48:19] aude: About? [14:51:00] (03CR) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 (owner: 10Paladox) [14:54:27] (03PS22) 10Paladox: [CheckUser] Move jshint to check: and use npm for jshint [integration/config] - 10https://gerrit.wikimedia.org/r/225182 [14:56:36] (03CR) 10Paladox: Update Maintenance extension tests [integration/config] - 10https://gerrit.wikimedia.org/r/225222 (owner: 10Paladox) [14:56:42] (03PS10) 10Paladox: Update Maintenance extension tests [integration/config] - 10https://gerrit.wikimedia.org/r/225222 [14:58:17] Krenair: Heh, "this should be ok now" from July 30th :p [14:58:20] https://gerrit.wikimedia.org/r/#/c/226971/ [14:58:23] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #385: ABORTED in 3.2 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/385/ [14:58:27] (03CR) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 (owner: 10Paladox) [15:07:29] (03CR) 10Paladox: "This is now ready to merge." [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [15:08:40] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #386: STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/386/ [15:11:12] 6Release-Engineering-Team, 10Wikimedia-Apache-configuration, 6operations: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#1666506 (10demon) [15:12:03] twentyafterfour: I just removed Possible-Tech-Projects from that ^ and added us instead. [15:12:31] Only reason it was listed there was because someone had planned to work on it at the hackathon at some point and those tasks got mass-turned into possible outreach projects afaict. [15:13:22] 6Release-Engineering-Team, 10Wikimedia-Apache-configuration, 6operations: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#1666511 (10mmodell) Probably duplicate of T104352 [15:13:53] ostriches: smart move not touching that one :) I just abandoned it for now cc Krenair [15:14:35] ostriches: should we add #deployment-systems to that last one you touched? [15:14:45] ah, mukunda is getting ahead of me [15:14:53] * greg-g catches up, still on first cup o' coffee [15:15:50] (03PS8) 10Hashar: Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:16:02] (03CR) 10Hashar: "rebased" [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:17:05] 6Release-Engineering-Team, 10Wikimedia-Apache-configuration, 6operations: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#1666526 (10demon) Actually, I think that depends on this. It asks for an API to use, and this task proposes... [15:17:09] (03CR) 10Paladox: "Should be jshint not jslint." [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:17:27] (03CR) 10jenkins-bot: [V: 04-1] Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:18:07] 10Deployment-Systems, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, and 4 others: is shown instead of the actual message on Special:ContentTranslation in ... - https://phabricator.wikimedia.org/T112964#1666530 [15:18:11] (03PS9) 10Paladox: Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 [15:18:33] (03CR) 10Paladox: "Sorry wrong one it should be jsonlint not jslint." [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:19:38] (03CR) 10Hashar: "my bad :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:21:00] (03CR) 10Hashar: [C: 032] Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:21:16] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:22:33] (03Merged) 10jenkins-bot: Update TwitterLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:23:42] thcipriani: so yeah force merges would cause havoc on Zuul :( [15:23:47] typical case: https://gerrit.wikimedia.org/r/#/c/139696/ [15:23:49] a test job fail [15:24:01] but the result is overriden and patch force merged :( [15:26:11] and the symptoms of this problem are zuul stops processing the queue? [15:29:06] 6Release-Engineering-Team: Gerrit clean up day - releng - https://phabricator.wikimedia.org/T113028#1666573 (10greg) As of right now (8:27am on Sept 23rd) these are the numbers for those searches (I think we already missed a few/bunch cleared by Chad/Alex this morning?): * [[ https://gerrit.wikimedia.org/r/#/q/s... [15:29:47] (03CR) 10Hashar: "Jobs pass https://gerrit.wikimedia.org/r/#/c/225713/" [integration/config] - 10https://gerrit.wikimedia.org/r/225712 (owner: 10Paladox) [15:30:12] thcipriani: yeah it has a deadlock somewhere [15:31:40] thcipriani: https://phabricator.wikimedia.org/T93812 (closed) [15:33:21] 6Release-Engineering-Team: Gerrit clean up day - releng - https://phabricator.wikimedia.org/T113028#1666582 (10greg) (and obviously everything Antoine/Zeljko did during their day) [15:35:30] greg-g: Just 5 between midnight and...now-ish? [15:35:42] ostriches: did you sleep? [15:36:06] About 4 hours. [15:36:12] 0130 - 0530 [15:36:18] geez [15:37:29] Also, those queries don't include operations/mw-config [15:37:33] Which is what I've been poking [15:37:47] they don't? grr, I thought I added that, maybe I made that up [15:38:30] (03CR) 10Chad: [C: 032] gitmailmap: generate a mailmap from USERINFO [tools/release] - 10https://gerrit.wikimedia.org/r/239343 (owner: 10Hashar) [15:38:46] (03Merged) 10jenkins-bot: gitmailmap: generate a mailmap from USERINFO [tools/release] - 10https://gerrit.wikimedia.org/r/239343 (owner: 10Hashar) [15:58:24] fwiw, I don't see much insanity in the zuul pipeline due to the cleanup.... yet: https://grafana.wikimedia.org/#/dashboard/db/releng-zuul [15:58:26] grmbbl [15:58:27] ;) [15:58:43] thcipriani: other idea to look at are jobs that have verified -1 https://gerrit.wikimedia.org/r/#/q/is:open+label:verified-1,n,z [15:59:24] the qunit jobs are broken :-(( [15:59:48] (03CR) 10Awight: "Ping. What are the rules about merging this repo, do you have to deploy immediately afterwards?" [tools/release] - 10https://gerrit.wikimedia.org/r/230705 (owner: 10Awight) [16:00:13] !log unpooling integration-slave-trusty-1014 and integration-slave-trusty-1017 . They fail both https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/ and https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/ [16:00:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:00:31] stupid new slaves [16:00:59] Yippee, build fixed! [16:01:00] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #10: FIXED in 4 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/10/ [16:01:37] (03CR) 10Paladox: "Ok done this can be merged now." [integration/config] - 10https://gerrit.wikimedia.org/r/225182 (owner: 10Paladox) [16:03:14] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:14] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:16] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:04:00] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1666667 (10Jdlrobson) 5Open>3Resolved Latest Jenkins build has passed https://integration.wikimedia.or... [16:04:21] (03PS4) 10Paladox: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 [16:04:27] (03PS5) 10Paladox: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 [16:04:42] (03CR) 10Paladox: "Re running jenkings." [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [16:05:03] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666673 (10hashar) [16:05:40] OAAHH [16:06:19] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666658 (10hashar) ``` 00:00:06.521 + curl --include http://localhost:9412/jenkins-mediawiki-core-qunit-49493/index.php/Special:BlankPage ... 00:00:06.567 PHP is not configured yeaahhhhh [16:07:30] 10Beta-Cluster, 10pywikibot-core, 5Patch-For-Review: Link.langlinkUnsafe does not work on Beta-Cluster wikis - https://phabricator.wikimedia.org/T112006#1666677 (10Krenair) 5Resolved>3Open [16:08:34] (03PS5) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/236258 [16:10:29] (03PS12) 10Paladox: Move MwEmbedSupport from deprecated jslint test to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/228488 [16:11:01] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666686 (10hashar) On integration-slave-trusty-1014 I created a test case file for http://localhost:9412/test.php : ``` lang=php PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [16:13:45] (03PS6) 10Paladox: Add extension-unittests-generic to metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/226995 [16:14:23] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666691 (10hashar) Will need a qunit job tied to one of the instance and try to repro. Maybe restarting Apache solved it. [16:17:35] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 8774 bytes in 0.531 second response time [16:19:11] (03CR) 10Greg Grossmeier: "FWIW: I haven't updated the wiki pages yet b/c I was waiting for the code changes to happen :)" [tools/release] - 10https://gerrit.wikimedia.org/r/239562 (https://phabricator.wikimedia.org/T67306) (owner: 10Florianschmidtwelzow) [16:21:36] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 8829 bytes in 1.202 second response time [16:22:38] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31807 bytes in 0.495 second response time [16:23:00] Already looking with Krenair ^ [16:23:08] that was my fault [16:23:28] I was debugging something on deployment-bastion and jenkins auto-deployed the changes [16:24:52] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666706 (10hashar) Repooled https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1014/ with no labels. Copy pasted the job to https://integration.wikimedia.... [16:26:38] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 46005 bytes in 1.266 second response time [16:27:21] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1666709 (10hashar) Seems it working. will pool them back tonight unless someone does. For trusty-1014 we need to adjust slaves. [16:28:13] will be back later [17:03:13] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 5Patch-For-Review, and 2 others: "Nonce already used" regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1666826 (10XZise) Not sure if that patch helped but since a few days all tests on Travis connecting to OAuth... [17:04:01] https://integration.wikimedia.org/zuul/ ... one of the jenkins gate-and-submit jobs seems stuck for an hour (parsoid) [17:09:53] legoktm can you help? [17:09:58] ^^ [17:30:31] 10Deployment-Systems, 3Scap3: Support smooth transitions from Trebuchet managed deploys - https://phabricator.wikimedia.org/T113107#1666937 (10dduvall) [17:30:33] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 should be able to deploy/rollback service config as part of deploy - https://phabricator.wikimedia.org/T109512#1666938 (10dduvall) [17:30:34] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 should support rollback - https://phabricator.wikimedia.org/T109514#1666935 (10dduvall) 5Open>3Resolved >>! In T109514#1660699, @Nemo_bis wrote: > Nice! This has proved to be useful in translatewiki.net too (see http://laxstrom.name/blag/2015/01/02/o... [17:30:58] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Allow batch_size to be configured for each deploy stage - https://phabricator.wikimedia.org/T112841#1666940 (10dduvall) 5Open>3Resolved [17:31:26] subbu: looks fixed now? sorry I was on a train [17:31:35] it also looks like we're low on trusty slaves? [17:31:58] legoktm, no problem. yes .. cscott fixed it. [17:33:50] subbu: what was the fix? [17:33:57] cscott, ^ [17:34:07] * subbu heads into sos [17:34:14] subbu: ahh, I see [17:34:19] cscott: nvm [17:34:24] * greg-g reads backscroll in -parsoid [17:39:16] (03PS1) 10Paladox: [TopTenPages] Update jenkings tests for this extension [integration/config] - 10https://gerrit.wikimedia.org/r/240425 [17:41:01] (03CR) 10Paladox: "I made a mistake in one of the patches and wasent aware because the json test wasent run because the test was added to the extension. this" [integration/config] - 10https://gerrit.wikimedia.org/r/240425 (owner: 10Paladox) [17:48:23] (03PS1) 10Paladox: [MassMessage] Add dependance on LiquidThreads [integration/config] - 10https://gerrit.wikimedia.org/r/240429 (https://phabricator.wikimedia.org/T113497) [17:48:40] (03PS2) 10Paladox: [MassMessage] Add dependance on LiquidThreads [integration/config] - 10https://gerrit.wikimedia.org/r/240429 (https://phabricator.wikimedia.org/T113497) [17:49:20] legoktm: We're very low on Trusty slaves in CI, yeah. :-( [17:50:05] (03CR) 10Legoktm: [C: 04-2] "This is wrong, and has nothing to do with the linked bug." [integration/config] - 10https://gerrit.wikimedia.org/r/240429 (https://phabricator.wikimedia.org/T113497) (owner: 10Paladox) [17:50:47] this will be a fun graph later today: https://grafana.wikimedia.org/#/dashboard/db/releng-zuul [17:58:14] 10Continuous-Integration-Config, 7Regression: [Regression] mwext-mw-selenium is slowing down developer productivity - https://phabricator.wikimedia.org/T113501#1667064 (10Krinkle) 3NEW [18:10:47] ostriches: pong [18:15:12] aude: Is https://gerrit.wikimedia.org/r/#/c/208655/ still needed/ok? [18:20:52] * aude looks [18:21:40] ostriches: it is needed but it's something we need to announce that we will be doing [18:21:48] mmk [18:21:53] might affect people like the pywikibot folks [18:22:03] I was just combing the mw-config backlog earlier and it looked trivial. [18:22:20] * aude reminded to poke lydia etc and get this announced and then do [18:28:41] (03PS1) 10Chad: Remove PHP localization cache code [tools/scap] - 10https://gerrit.wikimedia.org/r/240440 [18:28:57] 10Continuous-Integration-Config: Disable mediawiki-phpunit-*-composer jobs for older MediaWiki core branches - https://phabricator.wikimedia.org/T113506#1667229 (10Legoktm) 3NEW [18:28:57] thcipriani, marxarelli ^ [18:29:03] Eh, above by 1. [18:29:30] :P [18:29:40] are we giving up on the PHP localization cache? [18:31:48] https://gerrit.wikimedia.org/r/#/c/224629/ - last comment [18:31:58] I'm taking that as "yes, giving up" [18:32:35] :( [18:36:01] !log reapplying labels contintLabsSlave UbuntuTrusty phpflavor-hhvm on https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1014/ . https://phabricator.wikimedia.org/T113489 [18:36:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:40:54] 10Continuous-Integration-Infrastructure: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1667283 (10hashar) 5Open>3Resolved a:3hashar I have pooled them back in. The qunit jobs work. I guess the issue is puppet did not restart Apache after the PHP5 or qunit... [18:41:08] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance: New Trusty slaves can't run the mediawiki qunit jobs - https://phabricator.wikimedia.org/T113489#1667286 (10hashar) [18:48:43] RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0] [18:53:55] 10Continuous-Integration-Config, 7Regression: [Regression] mwext-mw-selenium is slowing down developer productivity - https://phabricator.wikimedia.org/T113501#1667337 (10Legoktm) [18:54:39] (03CR) 10Ori.livneh: [C: 04-1] "Thanks Chad. Looks like I am owed a slap for this :) However, the third change (If3bd73f6b4) should not be reverted -- that bit was succes" [tools/scap] - 10https://gerrit.wikimedia.org/r/240440 (owner: 10Chad) [18:54:42] 10Continuous-Integration-Config, 7Regression: [Regression] mwext-mw-selenium is slowing down developer productivity - https://phabricator.wikimedia.org/T113501#1667064 (10Legoktm) (Changed bullets to numbers to make them easier to reference) #2 seems like the best solution for now, though I'm still in favor o... [19:01:39] (03PS1) 10JanZerebecki: Don't run mediawiki-phpunit-*-composer on REL1_23 [integration/config] - 10https://gerrit.wikimedia.org/r/240447 (https://phabricator.wikimedia.org/T113506) [19:02:00] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down developer productivity - https://phabricator.wikimedia.org/T113501#1667382 (10hashar) p:5Unbreak!>3Normal [19:03:18] so uhm, this needs to go in soon (semantic versioning change) , but I don't want to merge it without some sort of review, so plz and thank you: https://gerrit.wikimedia.org/r/#/c/228039/ [19:04:55] (03PS1) 10Merlijn van Deen: Add fpm-based debian packaging [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 [19:05:30] (03CR) 10Merlijn van Deen: [C: 04-1] "This is one option; I'm still researching the option of backporting the sid package and how much work that would be." [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [19:06:39] (03CR) 10Merlijn van Deen: "Resulting .deb is https://tools.wmflabs.org/dpkg/composer_1.0.0-alpha10_all.deb" [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [19:19:27] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1667445 (10dduvall) [19:26:00] (03PS1) 10Paladox: Update squizlabs/php_codesniffer to 2.3.4 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 [19:27:15] (03PS1) 10Paladox: Update jakub-onderka/php-parallel-lint to 0.9.* [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240460 [19:30:43] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Page-Banner: test WikidataPageBanner its integration with Wikibase - https://phabricator.wikimedia.org/T106772#1667535 (10Jdlrobson) [19:41:23] (03CR) 10Polybuildr: [C: 032] Update jakub-onderka/php-parallel-lint to 0.9.* [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240460 (owner: 10Paladox) [19:50:43] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1667619 (10hashar) > The value that CI brings to developers is the fast feedback loop. If the slowdown of selenium is acceptable to the mobile team, that is their... [20:00:29] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1667639 (10hashar) For the non ranting part: Lets make the mwext-mw-selenium job faster, i.e. less than mediawiki-phpunit-zend which is 10 minutes. We have the... [20:04:48] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, 7Blocked-on-Operations: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1667658 (10greg) >>! In T100519#1655357, @greg wrote: >>>! In T100519#1529708, @BBlack wrote: >> Basically, yeah. I ran down a similar plan... [20:06:59] jdlrobson: seems mwext-mw-selenium is quite slow (since a 22 minutes run) [20:07:11] jdlrobson: there is a long task at https://phabricator.wikimedia.org/T113501 [20:07:29] jdlrobson: seems we might want to drop the @integration tag from some of the scenarii until they are made faster [20:07:36] 10Beta-Cluster, 10pywikibot-core, 7Pywikibot-tests: Search test failed - https://phabricator.wikimedia.org/T113517#1667681 (10XZise) The search seems to be broken/down: http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&generator=search&gsrsearch=wiki [20:08:12] (03CR) 10Paladox: "Thanks." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240460 (owner: 10Paladox) [20:09:43] 10Continuous-Integration-Infrastructure: mwext-mw-selenium job fails with: avconv not found on your system. - https://phabricator.wikimedia.org/T113520#1667693 (10hashar) 3NEW a:3hashar [20:10:25] twentyafterfour: i can take a crack at the git/ssh stuff here I think I understand what needs to happen...I think :D but git-ssh.wikimedia.org or vcs.wikimedia.org [20:10:27] or niether [20:10:29] greg-g: ^ [20:11:28] (03CR) 10Merlijn van Deen: "So... I built this in about half an hour, while the alternative (backporting stuff from sid) took me an hour without result (see https://e" [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [20:12:09] Hrm. [20:12:28] chasemp: Why does it have to be a separate vhost? [20:12:42] the long story short is phab.wm.o shares an IP with many other things [20:12:52] and the consensus was it was bad to put a new service on 22 only for phab there [20:12:55] so new ip for ssh [20:13:12] I'm...fine w/ it [20:13:17] 10Continuous-Integration-Infrastructure: mwext-mw-selenium job fails with: avconv not found on your system. - https://phabricator.wikimedia.org/T113520#1667716 (10hashar) On Trusty it is provided by `libav-tools`. Got installed manually (i.e. via explicitly via puppet): ``` hashar@integration-slave-trusty-1011:~... [20:13:34] chasemp: Bleh, misc-lb strikes again. [20:13:36] and there is some concerns about HA things re: this shared IP and portability for non http services across dc's [20:13:39] right [20:13:49] at one point I had a really detailed answer for this that now in my mind is [20:14:01] let's just get something going and that was a long tailed discussion to nowhere [20:14:20] if I set up a domain w/ an IP I can do a separate service in lvs and have that terminate to iridium [20:14:41] I basically understand all the compontents there but the domain itself I've been bikeshedding in my own brain [20:14:50] git-ssh wfm if we have to go that route. [20:15:27] Too bad git.wm.o is already overloaded :( [20:15:36] I wanted it yeah [20:15:44] I was kind of waiting for gitblit to die before [20:16:01] Now I want it dead. Like today. [20:16:42] 10Continuous-Integration-Infrastructure: mwext-mw-selenium job fails with: avconv not found on your system. - https://phabricator.wikimedia.org/T113520#1667741 (10hashar) So https://gerrit.wikimedia.org/r/#/c/234699/ //Use backported ffmpeg for multimedia transcoding on Trusty// That changes `mediawiki::package... [20:16:48] (03Merged) 10jenkins-bot: Update jakub-onderka/php-parallel-lint to 0.9.* [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240460 (owner: 10Paladox) [20:16:50] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 5Patch-For-Review, and 2 others: "Nonce already used" regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1667743 (10greg) a:5bd808>3None [20:17:25] (03PS2) 10Paladox: Update squizlabs/php_codesniffer to 2.3.4 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 [20:18:23] (03CR) 10Legoktm: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 (owner: 10Paladox) [20:19:20] chasemp: ostriches I think you two are good. I concur that git.w.o would have been great, too effing bad we couldn't kill it quicker [20:19:45] great I'll put up git-ssh.wm.o [20:19:51] and maybe we can come back someday [20:20:25] Watch me. [20:20:32] I'm not going to sleep until it's dead. [20:20:36] oh shit [20:20:42] * ostriches grabs another coffee [20:20:49] I just ruined chad's weekend [20:20:58] please don't start on meth [20:21:00] (03CR) 10Legoktm: "What's the context for this?" [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [20:21:07] Heh, I'm gone this weekend! I've got today, tomorrow and friday hehe [20:21:22] well we can always move git.wm.o to the same IP [20:21:23] later [20:21:42] * greg-g nods [20:21:54] ok in between my things I will try to set this in motion [20:22:02] what about "review.wikimedia.org"? [20:22:40] This is just the SSH bit for git operations, doesn't seem appropriate to me [20:22:55] ah [20:23:00] !log mwext-mw-selenium fails due to missing avconv https://phabricator.wikimedia.org/T113520 [20:23:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:23:07] well as long as its on port 22, I'm fine with everything :P [20:23:13] anything* [20:23:17] ha, that's the plan [20:23:29] yeah and git.wikimedia.org [20:23:42] so I can just git clone ssh://git.wikimedia.org/mediawiki/core.git ! [20:23:59] (sorry tongue-in-cheek- [20:24:23] that's the dream we are all dreaming [20:24:28] as sad as that may be [20:24:42] just s/gitblit/phab/ #done [20:30:59] chasemp: is the issue that git.wm.o points to gitblit and we need the ssh to go to phobia ? [20:31:00] s/phobia/phab/ [20:31:07] I say: lets shoot gitblit right now [20:31:37] I am not sure who uses it to be honest beside search engine bots and people unlucky enough to follow links from Gerrit [20:31:55] I thought the same thing and then ppl came out of the woodwork who rely on it [20:32:34] yeah, don't just do it, we'll need to announce it's demise [20:32:44] people will have to update git remotes if the pull from it [20:32:47] etc [20:32:58] you know, every change breaks someone's workflow [20:33:25] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: mwext-mw-selenium job fails with: avconv not found on your system. - https://phabricator.wikimedia.org/T113520#1667783 (10hashar) p:5Unbreak!>3Normal I have cherry picked the change and libav-tools has been installed: Notice: /Stage[main]/Con... [20:33:43] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1667785 (10greg) >>! In T113501#1667639, @hashar wrote: > Should I fill sub tasks for each? Yes please, let's keep things going forward positively. [20:33:59] no matter how broken or ugly it is someones favorite thing [20:33:59] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667786 (10zeljkofilipin) I do not know why mediawiki does not return zero banners when the header is sent. Anybody has an idea what is wrong? If nobody cares about this job, I vote... [20:35:08] 10Beta-Cluster, 10CirrusSearch, 6Discovery, 10pywikibot-core, 7Pywikibot-tests: Search test failed - https://phabricator.wikimedia.org/T113517#1667791 (10greg) [20:35:23] 10Beta-Cluster, 10CirrusSearch, 6Discovery, 10pywikibot-core, 7Pywikibot-tests: Search test failed - https://phabricator.wikimedia.org/T113517#1667794 (10greg) @ebernhardson, can you take a quick look? [20:36:08] PROBLEM - Puppet failure on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:38:25] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667809 (10zeljkofilipin) Looks like Selenium does not support adding custom headers, but some browser drivers support it, like phantomjs. Another alternative is to use a proxy like... [20:39:38] greg-g: how long would it take to setup a clone of beta labs? [20:40:10] jdlrobson: how long have you got? [20:40:16] I'm interested in accurately showing the impact of performance changes... I need some kind of `vagrant enable-role production` [20:40:16] jdlrobson: how long is a piece of string? [20:40:22] :D [20:40:33] I assume it relies on role unicorn [20:40:36] But I need to know [20:40:42] chasemp: ostriches: it is probably worth announcing that gitblit is going to be removed so people can adjust / prepare themselves [20:40:59] I'm struggling to work out how I can show a change X changes the eorld [20:41:15] jdlrobson: 4 men weeks and a few dozen of thousand dollars of hardware [20:41:34] hashar: That's what redirects are for. [20:41:34] jdlrobson: longer than you have [20:41:37] also beta clsuter [20:41:38] cluster [20:41:40] jdlrobson: several people talked about a performance cluster, but that seems unreasonable for us time/budget wise. [20:41:55] we spent 3 people's time for 1/2 of a quarter on that, didn't finish [20:42:03] That's not as bad as I thought actually. But yeh performance cluster is what I need.. [20:42:13] the way ops test performances [20:42:20] (03CR) 10Ejegg: [C: 031] "Necessary! The deployment branch should only be used for payments-wiki. Holding off +2 in case the rule is to deploy after each merge." [tools/release] - 10https://gerrit.wikimedia.org/r/230705 (owner: 10Awight) [20:42:30] is they unpool a machine, bring up on it whatever new version they want to test [20:42:38] then pool the machine back in and observe [20:42:51] cause at our load, lot of performance issues are only noticed with production traffic [20:43:04] so depends on what you want to test really. Performance team might have some clues [20:43:28] jdlrobson: this is the project to create a staging cluster: https://phabricator.wikimedia.org/tag/staging/ [20:43:29] (03PS14) 10Dduvall: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:43:41] jdlrobson: that's the state after 1.5 months work, as well :) [20:44:04] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: mwext-mw-selenium job fails with: avconv not found on your system. - https://phabricator.wikimedia.org/T113520#1667847 (10hashar) 5Open>3Resolved [20:44:23] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667850 (10jhobs) There is no longer anyone actively developing for Zero (as a main priority) and this looks like a trip down the same custom header rabbit hole I ran into a few mon... [20:44:59] (03CR) 10Dduvall: [C: 032] Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:45:12] Hmpff. [20:45:26] jdlrobson: once we do finish creating the staging cluster, the effort needed to make "yet another cluster" would be far less. Not one-click, but not a quarter worth of time :) [20:45:33] Is it possible to get a vagrant role that installs all our extensions or the majority of them? [20:45:36] there's a LOT to clean up about beta cluster [20:45:42] (All extensions on enwiki) [20:45:55] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:46:07] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [20:46:20] jdlrobson: also, next year, please encourage your managers to allow RelEng to not lose 1 headcount and instead gain another few people. We are routinely playing catchup. [20:46:42] on that subject if reading and performance team tell lila we need this does that speed up the quote? [20:46:43] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667869 (10hashar) >>! In T113463#1667809, @zeljkofilipin wrote: > Looks like Selenium does not support adding custom headers, but some browser drivers support it, like phantomjs. A... [20:46:57] cos i really think we do [20:47:06] nope [20:47:06] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667872 (10zeljkofilipin) I could not find a way to set custom headers for Chrome. I will take a look if it possible for Firefox tomorrow. [20:47:46] jdlrobson: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201516Year it's planned for Q3, everything between now and then are hard needs [20:48:17] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667894 (10hashar) >>! In T113463#1667850, @jhobs wrote: > There is no longer anyone actively developing for Zero (as a main priority) > and this looks like a trip down the same cus... [20:48:49] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667895 (10jhobs) I don't recall it being possible in Firefox either when originally working on these tests (as I'm sure I would've greatly favored that over PhantomJS), but I could... [20:48:52] unless lila gives RelEng our missing person (plus another) back, then we can't speed up any of those committments [20:49:41] jdlrobson: I think your best bet would be to write down in a task what you want to perf tests. then loop in Performance / Operations / Releng [20:49:48] i see. [20:49:53] jdlrobson: maybe it can be tested accurately on prod directly [20:50:07] if there is some harness to feature switch it on/off that should be reasonable [20:50:07] hashar: yet but the changes i want to do i need to test before they hit production [20:50:14] these are /big/ changes [20:50:19] so you would turn it on. Get traces. turn it off [20:50:36] 10Beta-Cluster, 10RESTBase, 6Services: Firewall rules too restrictive on deployment-restbase0x.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T113528#1667898 (10mobrovac) 3NEW [20:50:40] jdlrobson: maybe that can be done on test.wikipedia.org ? [20:50:41] (03CR) 10Merlijn van Deen: "Getting composer on tool labs. The other alternatives are" [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [20:50:43] i'm not expecting it will be that simple but we'll see [20:50:46] and yeah, I don't know if a separate, unused, cluster is a good way to test perf [20:50:49] yeh maybe test.wikipedia.org would be a good fit [20:50:56] we can deploy different branchs to test no? [20:50:59] and potentially [20:51:06] we could boot a dedicated app server on beta cluster for your test [20:51:07] (03CR) 10Merlijn van Deen: "and, possibly, this approach is interesting for contint as well?" [integration/composer] - 10https://gerrit.wikimedia.org/r/240451 (owner: 10Merlijn van Deen) [20:51:13] we did so for some security scanning [20:51:29] that might not take so long [20:51:38] jdlrobson: no branches [20:51:49] feature switches $wgFeatureAwesomeness = False [20:52:02] if $wgDBname == testwiki { $wgFeatureAwesomeness = True; } [20:52:12] hashar: why do integration/config jobs get backed up on the same gating queue (mediawiki)? [20:52:16] but for big changes that is not always possible :-( [20:52:32] 10Browser-Tests, 5Patch-For-Review: Fix or delete ZeroBanner browsertests Jenkins job - https://phabricator.wikimedia.org/T113463#1667906 (10jhobs) On further investigation, these tests wouldn't pass even if we did use PhantomJS (they're simply designed incorrectly). Definitely feel free to boot them. [20:52:32] marxarelli: they share a job with some repo that share a job with mediawiki/core [20:52:48] marxarelli: most probably the tox-py27 which is used on some mediawiki extensions and integration/config [20:52:55] 10Beta-Cluster, 10RESTBase, 6Services, 6operations: Firewall rules too restrictive on deployment-restbase0x.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T113528#1667909 (10greg) [20:52:59] marxarelli: and extensions and core share 'mediawiki-gate' (that one on purpose) [20:53:16] marxarelli: so zuul think that integration/config and mediawiki/core are dependent. We have a bug somewhere to split them off [20:53:34] hashar: so why could we not deploy a different feature branch? [20:53:42] technical restrictions or just policy? [20:53:52] marxarelli: from the discussion we had internally, maybe we want to prefix our jobs again [20:53:56] (03Merged) 10jenkins-bot: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:54:02] jdlrobson: both actually [20:54:33] jdlrobson: I am not sure how we would run different branches combinaison to fit all needs of the various dev teams. [20:54:45] jdlrobson: and on prod we enforce feature switching. So we can deploy the code, and turn it on later [20:54:54] hashar: would we be able to reserve test wiki for say a day? [20:54:58] e.g. a friday [20:55:07] and then switch it back to normal [20:55:08] hashar: is that the only way around the broad interdependency behavior of a dependent pipeline? [20:55:21] jdlrobson: unlikely cause beta cluster is widely used nowadays :-/ [20:55:34] test wiki or beta cluster or both? [20:55:59] jdlrobson: but as I said, write down a task, poke maybe ops list or qa list about it and us/others can figure out an appropriate solution for your needs! [20:56:08] marxarelli: yeah [20:56:08] jdlrobson: I'm still worried that testwiki won't give you the data you need (it gets zero actual use) [20:56:11] lemme grab the links [20:56:15] I guess it depends on what you're changing :) [20:56:22] marxarelli: Zuul consider two projects to be related when they share a job named the same :-/ [20:56:50] So this is what I'm trying to achieve -> https://phabricator.wikimedia.org/T113066 and https://phabricator.wikimedia.org/T112588 is the task making sure we show how kick ass our changes are [20:57:00] (03CR) 10Paladox: "Tests needs to be deployed on jenkings." [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:57:29] marxarelli: the Zuul v3 spec (equivalent to our RFC) mentions it http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#projects [20:57:34] marxarelli: they are switching to having queue being set explicitly [20:57:59] marxarelli: (and the jobs definition steps defined by Zuul itself as Ansible tasks that are sent to the slaves for execution) [20:58:24] !log Reloading zuul to deploy I4925da63b0aae22b3e809522466623d7958bfa24 [20:58:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:58:41] 10Beta-Cluster, 10CirrusSearch, 6Discovery, 10pywikibot-core, and 3 others: Search test failed - https://phabricator.wikimedia.org/T113517#1667938 (10EBernhardson) a:3EBernhardson [20:58:57] (03CR) 10Dduvall: "Sorry for the wait (long merge queues today). Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [20:59:40] https://grafana.wikimedia.org/#/dashboard/db/releng-zuul?panelId=10&fullscreen switch to the last 7 days :) [20:59:47] jdlrobson: my honest point of view is we can't do the performance impact of code. Though if we raised a few more million dollars and doubled the engineering team that is the kind of topic we could invest in [20:59:54] ebernhardson: thanks [20:59:58] hashar: to me, fixing this configuration is really what we should be focusing on re Krinkle's mwext-mw-selenium task [20:59:59] (03CR) 10Paladox: "Ok thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/226542 (owner: 10Paladox) [21:00:36] jdlrobson: what antoine said, actually. If you ask bd808 you'd get a similar response/reason: doing true perf testing that isn't just crap shoots requires prod hardware with prod users [21:00:51] * greg-g goes into a meeting [21:00:51] greg-g: okay another thought [21:00:54] the behavior of assuming such broad interdependency is just a false assumption [21:00:55] :) [21:00:59] jdlrobson: maybe that can be done by doing A/B testing and working closely with the Performance teams? They do have a lot of metrics [21:01:06] jdlrobson: I like you're thinking though, keep it coming :) [21:01:12] * greg-g goes for realzies [21:01:39] marxarelli: unless we extend mwext-mw-selenium to run all the @integration browser tests we have from all our repos :D [21:01:45] what if we created a new MobileFrontend extension, that worked on new.test.wikipedia.org similar to how MobileFrontend works and deployed that just there with similar mobile config [21:01:50] would that be plausible? [21:02:02] new mobilefrontend extension would just be a fork of mobilefrontend with some major tweaks [21:02:16] marxarelli: but yeah, we should probably split the dependent queue a bit now that the topic has been raised. [21:02:30] hashar: that's beside the point. the gating behavior needs fixing first [21:02:35] so test.m.wikipedia.org and test.new.wikipedia.org would be 2 separate entry points [21:03:10] marxarelli: though I would like to have those new splitted jobs to run on the Nodepool instance. One reason the jobs have been unified is that each of them fill a workspace on the permanent slaves and that quickly consumes all the disk space on them :-/ Because suddenly you have multiple copies / checkout of potentially huge repos [21:05:49] marxarelli: I am not sure how to split them to be honest :-/ [21:06:01] hashar: yeah, though i really don't understand why that was the right solution for disk space when some `find -mtime +15 -exec rm {} \;` cronjob would have sufficed [21:07:10] hashar: also, upping overall capacity isn't going to solve the gating issue since it's inherently a blocking operation [21:08:00] jdlrobson: probably too late to think about it for me :-/ sorry [21:08:26] marxarelli: the find -delete cron, I guess we haven't thought about it ::} [21:08:56] anyone want to take the relay baton from hashar and think about it with me? :) [21:09:09] marxarelli: we want to keep the .git repos in the workspace to avoid cloning them all over again. Though the mass delete could exclude .git dirs [21:09:17] hashar: :) oh well. let's discussion solutions to it soon [21:09:29] marxarelli: anyway, not going to be a problem on fresh instances :D [21:09:59] marxarelli: about the gating problem. I am not sure. We would need to figure out how to split the queues again without killing the perm slaves [21:10:29] marxarelli: so I really gotta write that doc explaining how to switch jobs to nodepool instances and start migrating them with a huge bucket ( "en masse" ? ) [21:10:52] jdlrobson: well we probably have no time to deal with it sorry :-/ [21:11:03] (03CR) 10Paladox: "Hi why is it wrong plus the extension uses code that is from liquidthread so in the test it needs to depend on the extension which is what" [integration/config] - 10https://gerrit.wikimedia.org/r/240429 (https://phabricator.wikimedia.org/T113497) (owner: 10Paladox) [21:11:39] jdlrobson: too many maintenance / project / different prio etc. But maybe some of releng can sneak at it and refine the idea, for the actual implementation it is unlikely we can provide anything anytime soon (think 6+ months) [21:12:23] (can't find a task) [21:13:58] (03PS13) 10Dduvall: Move MwEmbedSupport from deprecated jslint test to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/228488 (owner: 10Paladox) [21:17:26] (03CR) 10Dduvall: [C: 032] Move MwEmbedSupport from deprecated jslint test to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/228488 (owner: 10Paladox) [21:18:22] (03PS21) 10Dduvall: Add composer-test to interwiki [integration/config] - 10https://gerrit.wikimedia.org/r/225313 (owner: 10Paladox) [21:18:54] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/228488 (owner: 10Paladox) [21:19:01] 10Continuous-Integration-Config, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1668040 (10hashar) Related: {T101908} | https://gerrit.wikimedia.org/r/#/c/217188/ There is another one to get integration/config out of the main queue. Could n... [21:19:37] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: MultimediaViewer thumbnailBeforeProduceHTML hook breaks other extensions parser tests - https://phabricator.wikimedia.org/T69302#1668043 (10Jdlrobson) [21:19:48] marxarelli: also I had a #together moment with thcipriani earlier today. He is interested in learning more about jenkins / integration/config changes [21:19:59] seems your TZ overlaps more nicely [21:20:16] I don't mind having another pairing dedicated to that though (with anyone interested/asking) [21:20:24] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #11: FAILURE in 4 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/11/ [21:22:10] (03PS6) 10Paladox: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 [21:22:19] hashar speaks truth, marxarelli if you've got some time to run through integration/config gerrit cleanup #together, I'd be up for it. [21:23:36] (03PS10) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 [21:23:41] there is a lot of changes proposed by Paladox [21:23:46] he is doing nicely [21:24:00] but it is a bit tedious cause we have to look at the repo making sure composer/npm pass properly [21:24:06] deploy the npm/composer jobs [21:24:23] then +2 the patch on the source repo that adds composer/npm entry points [21:24:32] that should probably be automatized :-d [21:26:00] hashar: Paladox seems to be around and responding so i'm leaving the repo tests up to him/her [21:26:17] thcipriani: yeah, ring me if you seek #togetherness [21:26:34] marxarelli: yeah unfortunately he doesn't seem to run the npm/composer test commands :D [21:26:59] I should poke Timo to get him whitelisted and have jobs running (Timo vetoed it) [21:28:04] I still recommend against it. [21:28:45] Running into serious misunderstandings and problems everyday. It's a risk to our infrastructure. [21:29:18] crap it seems non-whitelisted users have gerrit merge rights which defeats explicitly not whitelisting them [21:30:10] that was probably me adding a job to the repo, without checking who all had merge rights [21:30:11] Krinkle: yeah I noticed that when reviewing his changes today. Though there are a lot of improvement compared to last summer [21:31:16] jzerebecki: yup that is rather inconsistent :D [21:31:23] so the question is do I now remove the jobs? [21:31:29] Maybe, but it's mostly luck. He's too impulsive with copying things from untrusted sources and hoping for the best without being able to why. [21:31:35] do we need to audit all repos? [21:32:08] jzerebecki: what are you talking about? [21:32:44] Krinkle: there is a repo where Paladox has merge rights which riggers in gate jobs that are reserved to whitelisted users. [21:33:29] I am not too worried [21:33:58] Yeah, that can only be abused if he explicitly commits malice to try and escape out of that workspace. But yeah, this does touch grey area. [21:34:23] I would say it is a limited scope of action [21:34:35] anyway [21:34:47] gonna migrate a bunch of tox / python jobs to nodepool instances tomorrow [21:34:59] and write down the step-by-step to migrate others [21:35:08] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, 7Blocked-on-Operations: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1668100 (10chasemp) [21:35:52] Krinkle: In Nodepool, think I will get zuul installed from our Gerrit. I get the lame bash foreach loop to clone the gerrit repos in the image already [21:36:09] 10Continuous-Integration-Infrastructure, 10Differential, 10Gerrit-Migration, 3releng-201516-q1: [keyresult] Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1668106 (10greg) Just talked with Mukunda about this one: he's done a lot of the prototyping but is getting befu... [21:36:20] Krinkle: going to grab the bits from your PHP script and copy paste the list of repos to clone in. Seem we can live with disk images of 3-4Gb, that should cover most use cases [21:36:24] hashar: Don't re-invent the wheel though. A basic loop afaik is not sufficient. [21:36:31] Krinkle: yeah I noticed [21:36:32] okay [21:36:41] Krinkle: might end up importing your PHP script :D [21:36:58] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, 7Blocked-on-Operations: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1668113 (10greg) a:3chasemp [21:37:10] Krinkle: will be for next week [21:37:17] hashar: It'd be good to add caches to normal slaves though. We're still a fair distance away from the majority of jobs running in nodepool. [21:37:25] There's tasks for it and known blockers. [21:37:29] thursday / friday is migrating a few tox / python jobs [21:37:30] It would also make them behave more similar. [21:37:35] yup [21:37:56] Did the Zuul patch for the hardlink thing get merged finally? [21:38:16] potentially we could build the mirror for the biggest repos right now, under /srv/jenkins-workspace somewhere and owned by jerkins-deploy user (for hardlink to work) [21:38:29] no the hardlink patch is still pending upstream [21:38:40] but should be cherry pickable in the .deb package I use to deploy zuul-cloner [21:38:50] though I am most probably going to abandon the .deb packaging [21:38:59] it is a pain in the ass to maintain [21:39:14] so instead of the labs slaves we could just pip install from gerrit integration/zuul @some branch [21:39:20] so a question, doing my reviews today i came across two patches that cherry pick a fix into the REL1_23 and REL1_24 branches. can i just merge to those for reasonable fixes? (https://gerrit.wikimedia.org/r/#/c/153565/) [21:39:23] and have pip install all the dependencies in some env [21:40:14] ostriches: ^ ebernhardson question [21:41:16] hashar: oh, I have a pip question for you [21:41:18] pip/venv [21:42:06] (03Merged) 10jenkins-bot: Move MwEmbedSupport from deprecated jslint test to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/228488 (owner: 10Paladox) [21:42:24] hashar: https://integration.wikimedia.org/ci/job/operations-puppet-tox-py27/10153/console is failing because the python-ldap library isn't installed, except it is installed via a system package on the CI slaves I manually logged into [21:42:25] ebernhardson: Already landed in master? Yeah merge away. [21:42:54] 10Continuous-Integration-Infrastructure, 7WorkType-NewFunctionality, 7Zuul: On slaves, install zuul-cloner from Gerrit in a venv instead of using a deb package - https://phabricator.wikimedia.org/T113538#1668164 (10hashar) [21:43:17] ostriches: sweet [21:43:29] legoktm: I know some deps can't be installed from pypi. pywikibot/core had such occurence [21:43:45] !log Reloading zuul to deploy I02685b2f0f4215904e4a364e2f8597d1ab47d657 [21:43:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:43:50] right...but it's already installed [21:44:44] on precise1014, pip show ldap [21:44:47] that yields nothing [21:44:53] but there is a python-ldap module [21:45:03] (03PS1) 10Paladox: [MwEmbedSupport] Add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/240594 [21:45:17] yeah [21:45:19] import ldap works [21:45:41] >>> import ldap [21:45:41] >>> ldap.__file__ [21:45:43] '/usr/lib/python2.7/dist-packages/ldap/__init__.pyc' [21:45:47] so that is provided by the .deb package [21:46:06] (03CR) 10Paladox: "This can be merged since the composer.json file is in repo." [integration/config] - 10https://gerrit.wikimedia.org/r/240594 (owner: 10Paladox) [21:46:20] legoktm: I guess the venv lacks it ? :( [21:46:27] (03Abandoned) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 (owner: 10Paladox) [21:47:13] greg-g: I added a few pros and one con to https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Gerrit_migration_why#Pros [21:47:16] legoktm: I can't remember whether tox import the system packages in the venv [21:47:21] hashar: does the venv override system packages? [21:47:23] ok :\ [21:47:35] (03CR) 10Paladox: "Hi has this change be deployed to jenkings." [integration/config] - 10https://gerrit.wikimedia.org/r/228488 (owner: 10Paladox) [21:48:00] legoktm: tox --help says: --sitepackages override sitepackages setting to True in all envs [21:48:03] legoktm: so I guess it doesn't import them by default [21:48:14] legoktm: and you have to add the module to the list of requirements [21:48:18] hrm [21:48:20] ok [21:48:40] kudos on owning change # 240000 :D [21:49:13] oh :P [21:49:19] legoktm: modules/admin/data/requirements.txt [21:49:39] legoktm: you will need to double check with ops to make sure python-ldap is installed on whatever machine is running that code [21:49:41] (03PS1) 10Dduvall: Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) [21:50:17] legoktm: and make sure the .deb package python-ldap is actually the same as the ldap module you rely on [21:50:29] hashar: ^ :) [21:50:34] temporarily [21:51:07] mhm [21:51:08] legoktm: can repo locally. After adding python-ldap I end up with: ImportError: No module named ldapsupportlib [21:51:16] marxarelli: :-}))))) [21:51:26] hashar: that's a relative import, I had to hack sys.path for it [21:51:59] marxarelli: https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/5490/console [21:52:04] marxarelli: it is apparently not changing the queues :-/ [21:52:12] marxarelli: oh no forget me [21:54:08] (03CR) 10Hashar: [C: 031] "Yeah that would no more run mwext-mw-selenium in the gate. Probably the easiest way to solve the delay reported by Timo." [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) (owner: 10Dduvall) [21:54:29] marxarelli: looks fine probably. But then I am very tired so might miss something [21:56:53] marxarelli: just found out that was part of Timo proposal #2 (only use 'test' and 'postmerge'). But I obsessed about postmerge and missed the implicit "remove job from gate-and-submit" [21:56:54] hashar: can you add thcipriani to the integration gerrit acl? [21:57:30] twentyafterfour: thankya, sir [21:57:32] ostriches: should we get other #releng folks to the Gerrit admin list ? [21:57:45] ostriches: so we can brea^Hassist in Gerrit maintenance #together [21:57:59] * hashar looks for ldap/releng [21:58:23] Which list? [21:58:38] !log Added marxarelli and twentyafterfour (wm.org emails) to the Gerrit 'integration' group [21:58:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:58:45] The admin group or the mail alias we barely use? [21:58:52] !log Added thcipriani to the Gerrit 'integration' group [21:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:59:09] ostriches: seems Dan can't add folks to Gerrit groups [21:59:18] but I am not sure which permission is needed for that [21:59:41] you need to be the owner of the group [21:59:44] maybe the Administrators one? https://gerrit.wikimedia.org/r/#/admin/groups/1,members [21:59:59] ahh [22:00:10] 'integration' is owned by "Project and Group Creators" [22:00:12] "Project and Group Creators" can add people to the integration group [22:00:28] https://gerrit.wikimedia.org/r/#/admin/groups/119,members [22:00:28] There's a bunch of groups with this ownership, it's messed up [22:00:34] yeah [22:01:19] there's 1040 groups though [22:01:27] might take a while for a gerrit admin to fix [22:01:34] (03CR) 10Legoktm: [C: 032] "I reviewed the output against core and didn't see any obvious regressions." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 (owner: 10Paladox) [22:01:43] Krenair: that can be scripted though [22:01:50] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, 7Blocked-on-Operations: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1668227 (10chasemp) [22:01:52] make them all self-owned? [22:02:00] (03Merged) 10jenkins-bot: Update squizlabs/php_codesniffer to 2.3.4 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 (owner: 10Paladox) [22:02:02] Krenair: no idea [22:03:59] (03CR) 10Dduvall: [C: 032] [MwEmbedSupport] Add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/240594 (owner: 10Paladox) [22:04:23] well [22:04:26] I am alseep [22:04:49] g'night :) [22:04:58] (03Merged) 10jenkins-bot: [MwEmbedSupport] Add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/240594 (owner: 10Paladox) [22:05:20] will not last long tomorrow [22:05:30] but got a bunch of patches merged at least! [22:05:53] !log added dan to gerrit admin group [22:05:53] (03CR) 10Paladox: "Thanks." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/240459 (owner: 10Paladox) [22:05:54] !log Reloading zuul to deploy I55c30dd85444f84d21972b39a6237bab86518f13 [22:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:06:05] !log added tyler to gerrit admin group [22:06:05] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/240594 (owner: 10Paladox) [22:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:06:15] marxarelli, thcipriani: Go drunk with power [22:06:59] greg-g: teaser/spoiler. Zuul emits Gerrit metrics to statsd. So there is a lame dashboard for Gerrit activity at https://grafana.wikimedia.org/#/dashboard/db/releng-gerrit :D [22:09:47] nice [22:18:01] (03PS2) 10Dduvall: Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) [22:19:22] ostriches: so in theory https://gerrit.wikimedia.org/r/#/c/240600/ makes the path open for SSH things to the public IP on iridium and it shoudl answer, 208.80.154.82/32 is now on lo on iridium [22:19:32] if you want to get your daemon stuff going to listen on that IP only for 22 [22:19:36] that's the next step [22:21:24] (03CR) 10Thcipriani: [C: 032] Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) (owner: 10Dduvall) [22:23:12] (03PS3) 10Thcipriani: Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) (owner: 10Dduvall) [22:24:30] (03CR) 10Thcipriani: [C: 032] Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) (owner: 10Dduvall) [22:25:19] (03Merged) 10jenkins-bot: Remove mwext-mw-selenium from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/240595 (https://phabricator.wikimedia.org/T113501) (owner: 10Dduvall) [22:27:18] (03PS7) 10Dduvall: Add Jenkins tests for DeletePagesForGood [integration/config] - 10https://gerrit.wikimedia.org/r/237138 (owner: 10Paladox) [22:38:56] !log Reloading Zuul to deploy I5db99dee0318b5854a05ea68870c57fb658f42e3 [22:38:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:55:56] Yippee, build fixed! [22:55:57] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #808: FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/808/ [22:57:27] (03CR) 10Thcipriani: [C: 032] Add extension-unittests-generic to metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [22:58:16] (03PS7) 10Thcipriani: Add extension-unittests-generic to metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:02:56] (03CR) 10Paladox: "Need c+2 again please." [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:03:34] (03CR) 10Thcipriani: [C: 032] Add extension-unittests-generic to metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:06:59] (03CR) 10Thcipriani: [C: 032] "Removing myself, re-adding, re +2ing" [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:11:02] (03CR) 10Paladox: "Thankyou?" [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:11:12] (03Merged) 10jenkins-bot: Add extension-unittests-generic to metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/226995 (owner: 10Paladox) [23:26:49] !log Reloading Zuul to deploy Iaa33b2002e898d68fe78f236e225c6e4bac61679 [23:26:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:44:18] Project beta-scap-eqiad build #71302: FAILURE in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/71302/ [23:56:44] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Set up phpunit structure tests for MediaWiki skin repositories - https://phabricator.wikimedia.org/T68926#1668925 (10Jdlrobson) [23:57:15] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Set up phpunit structure tests for MediaWiki skin repositories - https://phabricator.wikimedia.org/T68926#718859 (10Jdlrobson) Seems like remaining tasks are related to zuul config. Please re-add projects and point me at Vector/Monobook patches if I'v...