[00:44:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3491749 (10Nuria) mmm.. i think the tests might need to initialize semantic [01:18:07] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:53:06] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:35:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10User-MarcoAurelio: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3489008 (10MaxSem) Can't we just run this script via cron and publish the results somewhere web accessible? [04:20:16] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #473: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/473/ [07:50:30] (03PS1) 10Hashar: Pass PY_COLORS for tox 2.0.0+ [integration/config] - 10https://gerrit.wikimedia.org/r/369604 (https://phabricator.wikimedia.org/T169602) [07:54:37] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/369604 (https://phabricator.wikimedia.org/T169602) (owner: 10Hashar) [08:01:02] (03PS2) 10Hashar: Pass PY_COLORS for tox 2.0.0+ [integration/config] - 10https://gerrit.wikimedia.org/r/369604 (https://phabricator.wikimedia.org/T169602) [08:04:10] (03CR) 10Hashar: [C: 032] "PY_COLORS=1 is to enable color output in tox when its running without a tty. tox 2.0.0 strip all env variables, so we gotta instruct it to" [integration/config] - 10https://gerrit.wikimedia.org/r/369604 (https://phabricator.wikimedia.org/T169602) (owner: 10Hashar) [08:07:12] (03Merged) 10jenkins-bot: Pass PY_COLORS for tox 2.0.0+ [integration/config] - 10https://gerrit.wikimedia.org/r/369604 (https://phabricator.wikimedia.org/T169602) (owner: 10Hashar) [08:07:20] (03PS1) 10Hashar: docker: use tox --notest when populating cache [integration/config] - 10https://gerrit.wikimedia.org/r/369605 [08:10:01] (03CR) 10Hashar: "The way tox works, that would stall the environments to whatever dependencies were available at the time. If a repo defines its dependenc" [integration/config] - 10https://gerrit.wikimedia.org/r/369605 (owner: 10Hashar) [08:32:02] !log Regenerating Nodepool jessie image to upgrade tox from 1.9.2 to 2.5.0 - T169602 [08:32:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:32:07] T169602: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602 [08:42:45] !log - T169602 [08:42:48] bah [08:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:42:49] T169602: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602 [08:44:05] !log Image snapshot-ci-jessie-1501662758 in wmflabs-eqiad is ready - T169602 [08:44:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:02:18] !log Regenerating Nodepool Jessie image from scratch to get rid of tox 1.9.2 installed under /usr/local - T169602 [09:02:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:02:22] T169602: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602 [09:05:50] PROBLEM - Puppet errors on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:08:36] PROBLEM - Puppet errors on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:12:25] PROBLEM - Puppet errors on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:55:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3492360 (10hashar) It is still occurr... [09:58:36] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:50] RECOVERY - Puppet errors on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:25] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10User-MarcoAurelio: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3489008 (10hashar) The interwiki map is exposed on [[ https://en.wikipedia.beta.wmflabs.org/wiki/Special:Interwiki | Special:Inter... [10:30:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10User-MarcoAurelio: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3492453 (10hashar) Or in short: https://en.wikipedia.beta.wmflabs.org/w/api.php?action=sitematrix&format=json Loop through the s... [10:34:21] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:51:05] !log Image snapshot-ci-jessie-1501670727 in wmflabs-eqiad is ready - T169602 [10:51:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:51:08] T169602: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602 [10:58:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10User-MarcoAurelio: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3492551 (10MarcoAurelio) I think there's a misunderstanding here :) The problem is that after updating m:Interwiki map, to get th... [11:06:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602#3492568 (10hashar) WARNING: apparently tox now defaults to use python3 which breaks a bunch of jobs :( [11:09:21] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:31:55] !log Image snapshot-ci-jessie-1501673225 in wmflabs-eqiad is ready T169602 [11:31:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:31:59] T169602: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602 [11:39:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602#3492693 (10hashar) 05Open>03Resolved a:03hashar The tox based jobs now use version 2.5.0 installed from pypi. [11:48:31] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:59:46] hashar: I wonder why cxserver on Beta is not updated even when cxserver/deploy patch is merged. [12:00:04] hashar: logstash says: https://logstash-beta.wmflabs.org/goto/f3581f41d0a469a5b845e59a58da23f4 nothing other than that (probably not related) [12:16:16] kart_: I have no idea . Aren't the services manually updated? [12:17:00] or maybe it is puppet refreshing them, but even then I doubt it would run scap deploy on them [12:17:04] Is it? :) It is updated when deploy repo is updated AFAIK. [12:17:49] on deployment sca01 the last one is from July 14th [12:20:55] kart_: so in short. I have no idea how the services get updated on deployment-sca* hosts :\ [12:22:40] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #481: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/481/ [12:23:33] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:24:38] hashar: ok. Not priority, but good to keep in sync with production. [12:58:12] (03PS1) 10Hashar: Process extensions and skins submodules [integration/config] - 10https://gerrit.wikimedia.org/r/369642 (https://phabricator.wikimedia.org/T130966) [12:58:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: generalize extension submodule handling - https://phabricator.wikimedia.org/T130966#3493100 (10hashar) a:03hashar [13:11:31] (03CR) 10Hashar: "I have updated the non voting jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/369642 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:16:44] (03PS2) 10Hashar: Process extensions and skins submodules [integration/config] - 10https://gerrit.wikimedia.org/r/369642 (https://phabricator.wikimedia.org/T130966) [13:23:33] (03CR) 10Hashar: [C: 032] Process extensions and skins submodules [integration/config] - 10https://gerrit.wikimedia.org/r/369642 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:24:33] (03Merged) 10jenkins-bot: Process extensions and skins submodules [integration/config] - 10https://gerrit.wikimedia.org/r/369642 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:26:08] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), and 2 others: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3493180 (10zeljkofilipin) > [] Clarify whether both the te... [13:28:19] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3493185 (10zeljkofilipin) @Rfarrand as early in the day as possible would be the best for me. As far as I am concerned, there is no need for live studio... [13:31:25] (03PS1) 10Hashar: Make some extensions jobs voting [integration/config] - 10https://gerrit.wikimedia.org/r/369647 (https://phabricator.wikimedia.org/T130966) [13:32:05] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, and 2 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3493210 (10zeljkofilipin) @Jdlrobso... [13:38:00] (03PS1) 10Hashar: Process git modules recursively [integration/config] - 10https://gerrit.wikimedia.org/r/369649 (https://phabricator.wikimedia.org/T130966) [13:40:12] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3493263 (10zeljkofilipin) @Aleksey_WMDE we are almost there, the job is passing. Un... [13:44:09] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, and 3 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3493302 (10zeljkofilipin) [13:45:30] (03CR) 10Paladox: [C: 031] Process git modules recursively [integration/config] - 10https://gerrit.wikimedia.org/r/369649 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:46:25] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #479: 04FAILURE in 2 min 24 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/479/ [13:47:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: generalize extension submodule handling - https://phabricator.wikimedia.org/T130966#3493317 (10hashar) [13:48:38] (03PS2) 10Hashar: Make some extensions jobs voting [integration/config] - 10https://gerrit.wikimedia.org/r/369647 (https://phabricator.wikimedia.org/T130966) [13:49:02] (03CR) 10Hashar: [C: 032] "That fix the build for the Widgets skin :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/369649 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:49:09] (03CR) 10Hashar: [C: 032] "\O/" [integration/config] - 10https://gerrit.wikimedia.org/r/369647 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:50:01] (03Merged) 10jenkins-bot: Process git modules recursively [integration/config] - 10https://gerrit.wikimedia.org/r/369649 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:50:24] (03Merged) 10jenkins-bot: Make some extensions jobs voting [integration/config] - 10https://gerrit.wikimedia.org/r/369647 (https://phabricator.wikimedia.org/T130966) (owner: 10Hashar) [13:51:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other: LinkSuggest2 test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T155773#3493325 (10hashar) [13:51:45] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other: PagesList tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154930#3493327 (10hashar) [13:51:50] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other: GooglePlaces tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154848#3493328 (10hashar) [13:51:51] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other: FlickrAPI test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154847#3493329 (10hashar) [13:51:54] 10Continuous-Integration-Config, 10Brickimedia, 10MediaWiki-Core-Tests, 10Refreshed: Skin Refreshed sub repo does not handled in test config - https://phabricator.wikimedia.org/T154806#3493330 (10hashar) [13:51:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: generalize extension submodule handling - https://phabricator.wikimedia.org/T130966#3493323 (10hashar) 05Open>03Resolved extensions and skins now have their submodules process recursively. That fixed Widgets and pr... [13:55:18] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#3493337 (10hashar) [13:55:21] 10Continuous-Integration-Config, 10Brickimedia, 10MediaWiki-Core-Tests, 10Refreshed: Skin Refreshed sub repo does not handled in test config - https://phabricator.wikimedia.org/T154806#3493335 (10hashar) 05stalled>03Open The Jenkins job now process submodules. The MediaWiki test fails to find `WikiFon... [13:56:11] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10hashar) [13:56:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Brickimedia, 10MediaWiki-Core-Tests, 10Refreshed: Skin Refreshed sub repo does not handled in test config - https://phabricator.wikimedia.org/T154806#3493341 (10hashar) 05Open>03Resolved a:03hashar One can follow up on {T115436} [13:59:22] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3493371 (10Aleksey_WMDE) Ok, thank you. [13:59:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Create a Jenkins job that runs Echo RSpec tests daily - https://phabricator.wikimedia.org/T171753#3493372 (10zeljkofilipin) a:05zeljkofilipin>03None [13:59:34] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3493373 (10zeljkofilipin) a:05zeljkofilipin>03None [13:59:45] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: For MediaWiki extensions, merge rubocop inside mwext-mw-selenium-jessie - https://phabricator.wikimedia.org/T164479#3493374 (10zeljkofilipin) a:05zeljkofilipin>03... [13:59:54] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3493375 (10zeljkofilipin) a:05zeljkofilipin>03None [14:00:09] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Reading-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3493378 (10zeljkofilipin) a:05zeljkofilipin>03None [14:01:51] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other: GooglePlaces tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154848#2926038 (10hashar) Submodules are now process (T130966). It now fails with 13:59:21 1) AutoLoaderTest::testAu... [14:03:58] hashar i know how to fix ^^ [14:04:18] paladox: please do :} [14:04:24] ok :) [14:04:55] hashar: https://gerrit.wikimedia.org/r/#/c/369655/ :) [14:05:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other: FlickrAPI test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154847#2926008 (10hashar) submodules are now process (T130966). Now fails with: 13:59:50 1) AutoLoaderTest::testAutoLoadConfig 13:59:50 Failed as... [14:05:45] paladox: flickrapi has the same issue https://phabricator.wikimedia.org/T154847 [14:05:59] hmm [14:06:07] missing classes from the autoloader i guess [14:06:12] (03CR) 10Zfilipin: "Looks good to me in general, with a couple of whitespace questions." (032 comments) [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [14:06:12] * paladox looks to see which ones [14:06:25] hashar: passes https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie-non-voting/628/console :) [14:09:43] hashar: https://gerrit.wikimedia.org/r/#/c/369656/ :) [14:09:58] AWESOME [14:10:14] thanks :) [14:10:28] we can switch googleplaces to not be non voting now [14:10:32] * paladox submits patch for that [14:10:45] paladox: I have a patch for that [14:10:46] already :) [14:10:51] oh :) [14:10:54] +1 to it [14:12:08] hashar FlickrAPI passes https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie-non-voting/630/console :) [14:13:03] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#3493440 (10Paladox) [14:13:05] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other, 10Patch-For-Review: GooglePlaces tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154848#3493438 (10Paladox) 05Open>03Resolved a:03Paladox [14:14:13] thanks [14:14:18] (03PS1) 10Hashar: Make some extension voting [integration/config] - 10https://gerrit.wikimedia.org/r/369659 (https://phabricator.wikimedia.org/T154847) [14:14:32] paladox: ^^^ :) [14:14:38] ah :) [14:14:51] all fixed after I have FINALLY made CI to process submodules for mediawiki extensions and skins ( https://phabricator.wikimedia.org/T130966 ) [14:14:52] (03CR) 10Paladox: [C: 031] ":)" [integration/config] - 10https://gerrit.wikimedia.org/r/369659 (https://phabricator.wikimedia.org/T154847) (owner: 10Hashar) [14:15:02] :) [14:15:18] (03CR) 10Hashar: [C: 032] Make some extension voting [integration/config] - 10https://gerrit.wikimedia.org/r/369659 (https://phabricator.wikimedia.org/T154847) (owner: 10Hashar) [14:15:45] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#3493460 (10Paladox) [14:15:47] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other, 10Patch-For-Review: FlickrAPI test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154847#3493458 (10Paladox) 05Open>03Resolved a:03Paladox [14:16:26] (03CR) 10Hashar: [C: 032] Make some extension voting [integration/config] - 10https://gerrit.wikimedia.org/r/369659 (https://phabricator.wikimedia.org/T154847) (owner: 10Hashar) [14:16:43] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10Paladox) [14:16:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other, 10Patch-For-Review: LinkSuggest2 test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T155773#3493462 (10Paladox) 05Open>03Resolved a:03hashar [14:17:15] (03Merged) 10jenkins-bot: Make some extension voting [integration/config] - 10https://gerrit.wikimedia.org/r/369659 (https://phabricator.wikimedia.org/T154847) (owner: 10Hashar) [14:17:24] lots of tasks resolved :) [14:17:31] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10Paladox) [14:17:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other, 10Patch-For-Review: PagesList tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154930#3493471 (10Paladox) 05Open>03Resolved a:03hashar [14:17:37] \O/ [14:17:52] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#3493477 (10hashar) [14:17:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other, 10Patch-For-Review: PaginateText extension: Tests failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154935#3493475 (10hashar) 05Open>03Resolved a:03hashar [14:18:25] :) [14:18:35] there are still 63 extensions having extension-unittests-non-voting :( [14:19:21] oh [14:19:42] im rechecking https://gerrit.wikimedia.org/r/#/c/362286/ for T154803 [14:19:45] T154803: ResourcesTest fails on Skin CustomPage - https://phabricator.wikimedia.org/T154803 [14:19:45] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10hashar) There are still 63 extensions marked with non voting jobs in CI :-( mediawiki/extensions/BookManager mediawiki/extensions/DonationInterface media... [14:20:39] we should merge some wikibase tests into one [14:21:45] yup :( [14:21:57] thanks for the patches above [14:22:45] your welcome :) [14:33:36] (03PS1) 10Hashar: Add a note about mw debug file name [integration/jenkins] - 10https://gerrit.wikimedia.org/r/369663 (https://phabricator.wikimedia.org/T50002) [14:45:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10VPS-Projects, and 2 others: a lot of beta cluster instances are not reachable over SSH - https://phabricator.wikimedia.org/T171174#3493557 (10fgiunchedi) [14:45:21] 10Beta-Cluster-Infrastructure, 10media-storage, 10Patch-For-Review, 10User-fgiunchedi: deployment-ms-beXX Duplicate declaration: Exec[swift_udev_reload] - https://phabricator.wikimedia.org/T171454#3493555 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi [14:50:17] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:24] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:55:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3493590 (10chasemp) ```/home/rush# sudo bash swap_stat.sh inet_gethost (1... [15:01:45] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3375235 (10Kghbln) Note that now "ImageMap" has to be inv... [15:06:09] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3493620 (10Kghbln) > Interestingly "GraphViz" is still au... [15:12:53] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:20:01] (03PS1) 10Hashar: Assert MediaWiki does not generate error logs [integration/config] - 10https://gerrit.wikimedia.org/r/369676 (https://phabricator.wikimedia.org/T50002) [15:20:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#3493653 (10hashar) a:03hashar [15:20:44] (03PS1) 10Giuseppe Lavagetto: Move some functionality from run.sh to the Dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/369677 [15:20:46] (03PS1) 10Giuseppe Lavagetto: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 [15:20:48] (03PS1) 10Giuseppe Lavagetto: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 [15:25:53] (03CR) 10Hashar: "Shell shell shell! I tried it locally and it seems to work. Maybe we can enable it on a single job at first then generalize ?" (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/369676 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [15:26:00] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3493673 (10Kghbln) >>! In T168738#3493620, @Kghbln wrote:... [15:30:53] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, and 3 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3493701 (10Niedzielski) [15:40:45] (03PS4) 10Bekicot: Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 [16:10:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: wikimedia-fundraising-civicrm fails with Call to a member function getDriver() on null in phar:///srv/jenkins-workspace/worksp... - https://phabricator.wikimedia.org/T171724#3493781 [16:51:24] (03PS1) 10Hashar: Switch to flake8 for linting [integration/consistency] - 10https://gerrit.wikimedia.org/r/369692 [18:21:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3494394 (10chasemp) A few thoughts on this phenom. I'm not sure if rabbi... [18:23:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3494400 (10chasemp) Also, not a terrible idea as we start forcing rabbit... [18:33:08] btw, if the pool of nodepool takes longer to refill, why don't we make it bigger actually? do we have not enough ressources at labs for that? [18:45:56] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:46:15] who's messing with beta puppet? :) [19:25:34] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3494659 (10mmodell) [19:34:36] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3494708 (10mmodell) [19:49:17] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3494767 (10mmodell) [19:52:00] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3494772 (10Paladox) p:05Lowest>03Normal I believe most of this is done now :). We just need to try a scap deploy to see if it works. [20:05:03] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3494845 (10mmodell) [20:05:46] twentyafterfour: Hm.. third week in a row the same Wikibase issue (1st: T164173: jobs causing db replag load, 2nd: attempt to fix 1 causing T171370, 3rd: attempt to fix 2 causing T172320) [20:05:46] T172320: Error in Wikibase/client/includes/Changes/InjectRCRecordsJob.php line 120: Bad value for parameter $params: $params['change'] not set. - https://phabricator.wikimedia.org/T172320 [20:05:46] T171370: ERROR: "LBFactory::getEmptyTransactionTicket: WikiPageUpdater::injectRCRecords does not have outer scope" - https://phabricator.wikimedia.org/T171370 [20:05:46] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173 [20:06:13] Krinkle: lovely [20:06:13] I'm glad we caught it early this time so that we're not rolling forward the third week in a row with a known regression. [20:06:25] (the last two were caught on friday or next monday) [20:06:49] this one was immediately obvious because of the assertions in the code [20:07:20] better than silent failure :) [20:07:32] Yeah [20:08:04] twentyafterfour: Also, if our job runner were to report to Logstash (and scap including that channel, and including one in its canaries) we'd be able to catch the 1st one too. [20:08:43] wait, jobrunner doesn't report to logstash? The errors I saw seemed to come from runjobs [20:09:00] twentyafterfour: Yes, runJobs != job runner != jobchron. sorry. [20:09:08] oh [20:09:20] the latter two report to stderr only, piped to local /var/log by init.d/systemd [20:09:21] * twentyafterfour doesn't fully understand the job queue system [20:09:45] runJobs.php is run from within mw context (tis' an http end point), so those errors are logged by "mediawiki" normally. [20:09:50] * Krinkle joins a meeting [20:09:50] bbl [20:10:21] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Fix or remove Blubber's node_modules optimization - https://phabricator.wikimedia.org/T171632#3494865 (10dduvall) p:05Triage>03High [20:17:15] !log Update mobileapps to 2d8e8f6 [20:17:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:34:05] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:47] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:47] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:44:31] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:08:03] !log Running rebuildall.php on beta ruwiki [22:08:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:09:24] 10Scap: Scap: keyholder Too many authentication failures - https://phabricator.wikimedia.org/T172333#3495259 (10thcipriani) [22:10:25] 10Release-Engineering-Team (Kanban), 10Scap: Scap: keyholder Too many authentication failures - https://phabricator.wikimedia.org/T172333#3495273 (10thcipriani) p:05Triage>03Normal a:03thcipriani [22:15:12] 10Release-Engineering-Team (Kanban), 10User-greg: Setup 15 MW-Vagrant USB sticks for Wikimania 2017 Hackathon - https://phabricator.wikimedia.org/T172334#3495286 (10greg) [22:21:53] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Regression, 10Upstream, 10User-mobrovac: Gerrit flickering on change summary page - https://phabricator.wikimedia.org/T155122#3495328 (10Paladox) gwtui is being removed so proposing this as declined since upstream are really no longer taking patches that... [22:22:53] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 3 others: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3495336 (10greg) [22:23:58] greg-g: Just checking the "Holding the train" page. I'm glad to see the mention of performance team there as exception to post-Thursday. We're working on trying to get more and better data from group0/group1, but at the moment we are indeed only likely to find things on Friday/Monday. [22:24:19] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Regression, 10Upstream: Gerrit flickering on change summary page - https://phabricator.wikimedia.org/T155122#3495340 (10mobrovac) 05Open>03declined Fine by me. [22:24:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:52] Krinkle: /me nods [22:28:02] Krinkle: edits welcome for clarity but glad you like :) [22:28:24] 10Deployment-Systems, 10Gerrit, 10ReleaseTaggerBot, 10WorkType-NewFunctionality: Deployment status indicator for gerrit patches - https://phabricator.wikimedia.org/T88136#1004730 (10Paladox) This may now be possible as polygerrit is built from js and html not java a lot more people can contribute to do thi... [23:18:28] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-MarcoAurelio, 10User-greg: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3495489 (10greg) 05Open>03Resolved a:03greg > Added MarcoAurelio to deployment-prep. You are now a... [23:21:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Nodepool: Investigate nodepool slow deletion - https://phabricator.wikimedia.org/T172229#3495501 (10greg) [23:59:59] twentyafterfour: heh, here i am, i almost totally missed by an hour