[01:00:44] 10Continuous-Integration-Infrastructure: Require indenting of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826816 (10Huji) [01:06:15] 10Continuous-Integration-Infrastructure: Require indenting of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826827 (10Huji) p:05Triage>03Lowest [01:06:21] 10Continuous-Integration-Infrastructure: Require indenting of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826816 (10Huji) a:03Huji [01:22:13] 10Continuous-Integration-Infrastructure: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826831 (10Huji) [01:38:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [03:34:56] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826857 (10Huji) Besides the patch I just submitted (for MW core), there are 94 more files that need to be dealt with. I will wait for this one to be merge... [04:49:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:39:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [06:24:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:48:45] Yippee, build fixed! [06:48:45] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #571: 09FIXED in 2 hr 8 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/571/ [06:54:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [07:13:01] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:27:42] (03CR) 10WMDE-leszek: [C: 031] Remove DataTypes extension from make-wmf-branch script (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/394286 (https://phabricator.wikimedia.org/T180455) (owner: 10Addshore) [08:54:11] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:29:30] (03PS2) 10Filippo Giunchedi: operations-puppet: install mtail [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) [09:29:55] (03CR) 10Filippo Giunchedi: "> -1 for changelog change needed." [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) (owner: 10Filippo Giunchedi) [09:48:02] (03PS1) 10Hashar: Drop Zuul template 'npm' [integration/config] - 10https://gerrit.wikimedia.org/r/397415 [09:53:57] (03CR) 10Hashar: [C: 032] Drop Zuul template 'npm' [integration/config] - 10https://gerrit.wikimedia.org/r/397415 (owner: 10Hashar) [09:55:04] (03Merged) 10jenkins-bot: Drop Zuul template 'npm' [integration/config] - 10https://gerrit.wikimedia.org/r/397415 (owner: 10Hashar) [09:57:38] (03CR) 10Hashar: [C: 032] [SwiftCloudFiles] Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/396467 (https://phabricator.wikimedia.org/T182384) (owner: 10Umherirrender) [09:57:52] (03CR) 10Hashar: [C: 032] [GitHub] Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/396466 (https://phabricator.wikimedia.org/T182231) (owner: 10Umherirrender) [09:58:49] (03Merged) 10jenkins-bot: [SwiftCloudFiles] Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/396467 (https://phabricator.wikimedia.org/T182384) (owner: 10Umherirrender) [09:59:01] (03Merged) 10jenkins-bot: [GitHub] Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/396466 (https://phabricator.wikimedia.org/T182231) (owner: 10Umherirrender) [09:59:57] !log github: archiving https://github.com/wikimedia/mediawiki-extensions-SwiftCloudFiles - T182384 [10:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:00:04] T182384: Archive extension/SwiftCloudFiles - https://phabricator.wikimedia.org/T182384 [10:00:43] !log github: deleted https://github.com/wikimedia/mediawiki-extensions-GitHub | T182231 [10:00:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:00:52] T182231: Archive the mediawiki/extensions/GitHub mirror on gerrit - https://phabricator.wikimedia.org/T182231 [10:02:34] (03CR) 10Hashar: [C: 032] Make GraphViz depend on ImageMap [integration/config] - 10https://gerrit.wikimedia.org/r/396093 (owner: 10Reedy) [10:04:11] (03Merged) 10jenkins-bot: Make GraphViz depend on ImageMap [integration/config] - 10https://gerrit.wikimedia.org/r/396093 (owner: 10Reedy) [10:30:21] 10Release-Engineering-Team (Someday), 10Deployments, 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#3827389 (10Joe) @tstarling AIUI we should be able to switch mwscript to use... [10:37:27] 10Continuous-Integration-Infrastructure (shipyard), 10PAWS, 10monitoring, 10Graphite: Purge labs graphite metrics of Docker ephemeral partitions - https://phabricator.wikimedia.org/T181476#3827403 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi @hashar got it! I've deleted the remaining metrics from... [10:44:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: Make DataTypes repository read-only - https://phabricator.wikimedia.org/T182566#3827435 (10WMDE-leszek) [11:18:08] (03PS1) 10Hashar: Docker for maps/kartotherian maps/tilerator [integration/config] - 10https://gerrit.wikimedia.org/r/397501 [11:43:14] (03PS2) 10Hashar: Docker for maps/kartotherian maps/tilerator [integration/config] - 10https://gerrit.wikimedia.org/r/397501 [11:45:34] hashar: re: https://phabricator.wikimedia.org/T180841 - we need to setup tables in DB. Can you point me some wiki page on how to deal with DB in Beta cluster? [11:45:43] (For Translate) [11:46:41] (03CR) 10Hashar: [C: 032] Docker for maps/kartotherian maps/tilerator [integration/config] - 10https://gerrit.wikimedia.org/r/397501 (owner: 10Hashar) [11:50:25] (03Merged) 10jenkins-bot: Docker for maps/kartotherian maps/tilerator [integration/config] - 10https://gerrit.wikimedia.org/r/397501 (owner: 10Hashar) [11:51:44] (03PS1) 10Hashar: Rename npm-test-trending-edits -> npm-test-librdkafka [integration/config] - 10https://gerrit.wikimedia.org/r/397529 [11:52:29] (03PS2) 10Hashar: Rename npm-test-trending-edits -> npm-test-librdkafka [integration/config] - 10https://gerrit.wikimedia.org/r/397529 [12:00:26] (03CR) 10Hashar: [C: 032] Rename npm-test-trending-edits -> npm-test-librdkafka [integration/config] - 10https://gerrit.wikimedia.org/r/397529 (owner: 10Hashar) [12:01:49] (03Merged) 10jenkins-bot: Rename npm-test-trending-edits -> npm-test-librdkafka [integration/config] - 10https://gerrit.wikimedia.org/r/397529 (owner: 10Hashar) [12:02:16] (03PS1) 10Hashar: Move change-propagation npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397530 [12:02:35] (03CR) 10Hashar: [C: 032] Move change-propagation npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397530 (owner: 10Hashar) [12:03:47] (03Merged) 10jenkins-bot: Move change-propagation npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397530 (owner: 10Hashar) [12:09:43] 10Scap, 10Operations, 10Packaging: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3827653 (10akosiaris) 05Open>03Resolved a:03akosiaris Package built for both jessie and trusty and uploaded on apt.wikimedia.org. Puppet change merged as well [12:09:48] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3827656 (10akosiaris) [12:10:05] 10Scap, 10Operations, 10Packaging: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3827657 (10mmodell) Awesome thank you! [12:17:18] (03PS1) 10Hashar: Migrate service-pipeline repo (mathoid) to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397532 [12:22:08] (03CR) 10Hashar: [C: 032] Migrate service-pipeline repo (mathoid) to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397532 (owner: 10Hashar) [12:22:16] (03CR) 10Hashar: [C: 032] "Passed https://integration.wikimedia.org/ci/job/mathoid-npm-node-6-docker/1/console :)" [integration/config] - 10https://gerrit.wikimedia.org/r/397532 (owner: 10Hashar) [12:22:55] Yippee, build fixed! [12:22:55] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #613: 09FIXED in 54 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/613/ [12:23:41] (03Merged) 10jenkins-bot: Migrate service-pipeline repo (mathoid) to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397532 (owner: 10Hashar) [12:53:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.30-release, and 2 others: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3827775 (10Addshore) >>! In T182336#3824369, @hashar wrote: > ``` > lang=python... [13:04:54] 10Continuous-Integration-Config: Jenkins isn't showing mediawiki-core-php70-phan-docker failure output - https://phabricator.wikimedia.org/T180630#3827808 (10Addshore) p:05Unbreak!>03High Changing from UBN to High, as this is not UBN, the errors are still visible in the console log. [13:17:41] 10Continuous-Integration-Infrastructure (shipyard): operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3827836 (10Addshore) >>! In T178620#3821439, @hashar wrote: > That image is now based on `docker-pkg`. Maybe that is faster now. We should be able to make thi... [13:20:12] 10Continuous-Integration-Infrastructure (shipyard): Create docker capacity dashboard similar to nodepool one - https://phabricator.wikimedia.org/T177686#3827841 (10Addshore) Summary for the last 6 months! great progress! {F11768575} [13:21:47] (03PS1) 10Hashar: Add back files: filter for mediawiki-core-npm-node-6 [integration/config] - 10https://gerrit.wikimedia.org/r/397541 [13:25:36] (03CR) 10Hashar: [C: 032] Add back files: filter for mediawiki-core-npm-node-6 [integration/config] - 10https://gerrit.wikimedia.org/r/397541 (owner: 10Hashar) [13:26:36] (03Merged) 10jenkins-bot: Add back files: filter for mediawiki-core-npm-node-6 [integration/config] - 10https://gerrit.wikimedia.org/r/397541 (owner: 10Hashar) [13:32:15] (03PS1) 10Hashar: Drop "npm run lint" from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/397543 [13:41:02] (03CR) 10Hashar: [C: 032] Drop "npm run lint" from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/397543 (owner: 10Hashar) [13:42:11] (03Merged) 10jenkins-bot: Drop "npm run lint" from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/397543 (owner: 10Hashar) [13:46:35] (03PS1) 10Dalba: Increase pywikibot-core-tox-doc-docker timeout to 6 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/397544 [13:49:36] (03CR) 10Dalba: "Example of builds failing with a 5 min timeout:" [integration/config] - 10https://gerrit.wikimedia.org/r/397544 (owner: 10Dalba) [13:55:31] (03PS1) 10Hashar: Remove unneeded variable in a Zuul test [integration/config] - 10https://gerrit.wikimedia.org/r/397549 [14:05:07] (03CR) 10Dalba: "- https://integration.wikimedia.org/ci/job/pywikibot-core-tox-doc-docker/95/console" [integration/config] - 10https://gerrit.wikimedia.org/r/397544 (owner: 10Dalba) [14:05:51] (03PS2) 10Dalba: Increase pywikibot-core-tox-doc-docker timeout to 10 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/397544 [14:09:58] (03CR) 10Hashar: [C: 032] Remove unneeded variable in a Zuul test [integration/config] - 10https://gerrit.wikimedia.org/r/397549 (owner: 10Hashar) [14:11:06] (03Merged) 10jenkins-bot: Remove unneeded variable in a Zuul test [integration/config] - 10https://gerrit.wikimedia.org/r/397549 (owner: 10Hashar) [15:22:27] deployment-prep folks: I'm going to upgrade your puppetmaster from v3 to v4 in an hour or so. Past experience suggests that it will be largely painless, but it might cause some failed puppet runs during the transition. [16:41:06] Project beta-code-update-eqiad build #184783: 15ABORTED in 8.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/184783/ [16:41:14] Project beta-scap-eqiad build #185871: 15ABORTED in 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185871/ [16:44:25] Project beta-scap-eqiad build #185872: 04FAILURE in 0.44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185872/ [16:45:08] Project beta-scap-eqiad build #185873: 04STILL FAILING in 0.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185873/ [16:50:09] ^^ those are me [16:51:02] Project beta-scap-eqiad build #185874: 04STILL FAILING in 0.52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185874/ [16:53:38] Project beta-scap-eqiad build #185875: 04STILL FAILING in 0.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185875/ [16:57:07] i will be a little late [17:01:15] (03PS1) 10Hashar: docker: add support to override 'npm test' [integration/config] - 10https://gerrit.wikimedia.org/r/397576 [17:01:46] Yippee, build fixed! [17:01:47] Project beta-scap-eqiad build #185876: 09FIXED in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185876/ [17:02:58] hashar: See e-mail, we're starting ~10m late anyway [17:08:05] 10Continuous-Integration-Infrastructure (shipyard): operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3828577 (10hashar) Seems like as `root` it: * creates `/srv/workspace` * git clone puppet.git at `/srv/workspace/puppet` * install tox and bundle dependencies... [17:17:05] (03PS1) 10Hashar: docker: provision operations-puppet as 'nobody' [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) [17:18:30] 10Release-Engineering-Team (Kanban), 10Epic: FY2017/18 Program 3 Outcome 2 Objective 1: Define a “Technical Debt Project Manager” role that regularly communicates with all Foundation engineering teams regarding their technical debt - https://phabricator.wikimedia.org/T174093#3828615 (10Jrbranaa) 05Open>03Re... [17:18:32] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 2: Organizational technical debt is reduced. - https://phabricator.wikimedia.org/T174089#3828616 (10Jrbranaa) [17:19:09] (03PS2) 10Hashar: docker: provision operations-puppet as 'nobody' [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) [17:19:25] (03CR) 10Hashar: "rebased on tip of master" [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) (owner: 10Hashar) [17:23:14] (03PS2) 10Hashar: docker: add support to override 'npm test' [integration/config] - 10https://gerrit.wikimedia.org/r/397576 [17:23:48] (03CR) 10Hashar: [C: 032] docker: add support to override 'npm test' [integration/config] - 10https://gerrit.wikimedia.org/r/397576 (owner: 10Hashar) [17:24:44] !log Update node-6-docker jobs so the entry point recognizes setting NPM_RUN_SCRIPT=doc | https://gerrit.wikimedia.org/r/#/c/397576/ [17:24:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:25:22] (03Merged) 10jenkins-bot: docker: add support to override 'npm test' [integration/config] - 10https://gerrit.wikimedia.org/r/397576 (owner: 10Hashar) [17:42:26] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release Pipeline (Blubber): Establish blubber config in mathoid - https://phabricator.wikimedia.org/T173127#3519160 (10Physikerwelt) Is there a step by step guide how to test mathoid with blubber. I found https://github.com/marxarelli/blubber, but I... [17:54:54] 10Continuous-Integration-Config, 10MediaWiki-SecurityCheckPlugin, 10Security-Team: Make jenkins run security-check-plugin non-voting - https://phabricator.wikimedia.org/T182599#3828725 (10Legoktm) [17:59:03] (03PS1) 10Hashar: Do not look at this [integration/config] - 10https://gerrit.wikimedia.org/r/397583 [17:59:25] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#3828751 (10Umherirrender) [18:00:42] 10Release-Engineering-Team, 10Deployments, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820084 (10demon) We could move test2 to group1 and use that. Easiest solution, plus would get some extra traffic. [18:00:51] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10Umherirrender) [18:01:24] 10Continuous-Integration-Config, 10TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10Umherirrender) [18:01:33] 10Release-Engineering-Team (Kanban), 10Deployments, 10HHVM, 10Wikimedia-Incident: Figure out why HHVM kept running stale code for hours - https://phabricator.wikimedia.org/T181833#3828765 (10greg) [18:07:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday): Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes - https://phabricator.wikimedia.org/T93414#3828783 (10Legoktm) [18:10:07] is there a reason the RSS feeds on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/ are blocked? [18:11:50] Only on that job? [18:19:31] I didn't really check any other job [18:19:46] https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs-docker/rssFailed doesn't work [18:19:58] GRR_NPM_XXXXz: I feel your pain [18:25:51] The root ones on the first landing page do at least [18:30:18] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:30:23] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:30:55] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:31:13] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:31:47] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:32:10] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:32:22] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:33:40] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:33:50] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:34:10] That took a while :P [18:34:36] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:36:05] PROBLEM - Puppet errors on deployment-eventlog02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:37:01] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:38:01] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:39:22] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:40:08] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:40:08] PROBLEM - Puppet errors on deployment-memc07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:40:22] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:40:27] PROBLEM - Puppet errors on deployment-videoscaler01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:40:39] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:40:49] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:40:55] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:40:55] PROBLEM - Puppet errors on deployment-cumin is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:41:09] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:41:13] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:41:27] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:42:14] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:42:30] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:43:18] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:43:18] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:43:20] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:43:52] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:44:20] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:44:34] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:45:35] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:45:51] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:46:03] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:46:12] https://tools.wmflabs.org/coverage/mediawiki/chart.png [18:46:27] now to figure out how to backfill that even further... [18:46:35] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:47:48] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:49:51] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:50:15] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:50:41] PROBLEM - Puppet errors on deployment-redis05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:50:47] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:51:16] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:51:28] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:51:36] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:51:40] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:51:58] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:53:10] PROBLEM - Puppet errors on deployment-poolcounter04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:53:10] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:53:39] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:53:47] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:54:03] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:54:23] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:54:35] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:54:41] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:55:21] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:55:31] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:56:42] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:56:56] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:57:18] That noise is all me, it should all recover shortly [18:57:55] thanks andrewbogott [18:58:14] I tried to silence shinken pre-emptively but now I see that I had a typo in my commandline :( [18:58:37] Anyway, your puppetmaster is upgraded to v4 now and all should be well. Let me know if you see bad behavior (other than the latent shinken warnings). [18:59:00] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:00:03] GRR_NPM_XXXXz hi, with the recent ci changes that were merged not that long ago, im getting this error [19:00:04] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Class[Contint::Packages::Ruby] is already declared; cannot redeclare at /etc/puppet/modules/profile/manifests/ci/browsertests.pp:5 at /etc/puppet/modules/profile/manifests/ci/browsertests.pp:5:5 on node jenkins-slave-01.git.eqiad.wmflabs [19:00:04] Warning: Not using cache on failed catalog [19:00:04] Error: Could not retrieve catalog; skipping run [19:00:06] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:00:49] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:03:38] (silencing shinken for a few minutes so the recoveries aren't so noisy) [19:08:30] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3829022 (10mmodell) @zeljkofilipin would it make sense to put them in the... [19:09:05] then i get [19:09:05] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Class[Contint::Browsers] is already declared; cannot redeclare at /etc/puppet/modules/profile/manifests/ci/browsertests.pp:9 at /etc/puppet/modules/profile/manifests/ci/browsertests.pp:9:5 on node jenkins-slave-01.git.eqiad.wmflabs [19:09:05] Warning: Not using cache on failed catalog [19:09:06] Error: Could not retrieve catalog; skipping run [19:10:34] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829035 (10mmodell) @awight: can you try it now with the -v flag, scap should include the ssh logs in verbose output now. [19:11:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Enable Xdebug on Jenkins to calculate code coverage. - https://phabricator.wikimedia.org/T178701#3829039 (10Legoktm) [19:12:05] paladox: Related to puppet upgrades? [19:12:14] Reedy doin't think so [19:12:19] i havent upgraded puppet [19:12:22] still on 3.x [19:20:52] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829075 (10awight) Done. Logs are in scap-sync-2017-12-09-0004.log (sic., note that the dates are still misleading) Thanks for t... [19:21:56] twentyafterfour: ^ fyi. Nice patch btw! [19:23:04] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:14] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:23:32] RECOVERY - Puppet errors on deployment-mediawiki07 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:50] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:50] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:05] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:06] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:10] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:16] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:31] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:49] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:53] RECOVERY - Puppet errors on deployment-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:01] RECOVERY - Puppet errors on deployment-eventlog02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:22] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:23] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:25] RECOVERY - Puppet errors on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:28] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:30] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:30] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:32] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:38] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:38] RECOVERY - Puppet errors on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:38] Welp [19:26:00] RECOVERY - Puppet errors on deployment-redis05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:04] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:08] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:16] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:16] RECOVERY - Puppet errors on deployment-memc07 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:22] RECOVERY - Puppet errors on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:27] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:41] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:43] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:49] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:53] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:01] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:07] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:13] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:22] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:23] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:29] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:31] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:47] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:54] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:54] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:00] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:04] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:12] RECOVERY - Puppet errors on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:34] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829109 (10mmodell) Something really strange is going on... ``` debug1: Offering RSA public key: /etc/keyholder.d/deploy_servic... [19:28:38] 10Release-Engineering-Team, 10Deployments, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3829110 (10daniel) @demon I'm all for it! [19:30:04] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:17] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:28] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:29] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:29] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:43] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:29] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:42] RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:07] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:40] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:48] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:19] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:29] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:46:29] 10Scap, 10Services (watching): Scap reports a check failure as an SSH connection failure - https://phabricator.wikimedia.org/T182623#3829159 (10mobrovac) p:05Triage>03High [19:47:07] 10Scap, 10Services (watching): Scap reports a check failure as an SSH connection failure - https://phabricator.wikimedia.org/T182623#3829170 (10mobrovac) [19:53:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [19:58:53] no_justification upstream have finished notedb now. Google no longer use reviewdb. So i presume they will removed the support for reviewdb now. [19:59:10] https://groups.google.com/forum/#!topic/repo-discuss/UFEPwEs-leU [20:20:53] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:26:04] (03PS2) 10Hashar: Migrate VisualEditor 'npm doc' job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397583 [20:27:38] (03CR) 10Hashar: [C: 032] Migrate VisualEditor 'npm doc' job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397583 (owner: 10Hashar) [20:29:06] (03PS3) 10Hashar: Migrate VisualEditor 'npm doc' job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397583 [20:29:26] (03CR) 10Hashar: [C: 032] Migrate VisualEditor 'npm doc' job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397583 (owner: 10Hashar) [20:30:55] (03Merged) 10jenkins-bot: Migrate VisualEditor 'npm doc' job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/397583 (owner: 10Hashar) [20:46:31] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:48:53] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [21:04:02] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release Pipeline (Blubber): Establish blubber config in mathoid - https://phabricator.wikimedia.org/T173127#3829337 (10thcipriani) >>! In T173127#3828686, @Physikerwelt wrote: > Is there a step by step guide how to test mathoid with blubber. I found... [21:06:55] !log deployed mobileapps@61ca333 to BC [21:07:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:31:31] (03PS2) 10Hashar: contint::browsers is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392977 [21:32:00] (03PS3) 10Hashar: dib: contint::browsers is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392977 [21:41:50] (03CR) 10Hashar: [C: 032] dib: contint::browsers is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392977 (owner: 10Hashar) [21:43:21] (03Merged) 10jenkins-bot: dib: contint::browsers is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392977 (owner: 10Hashar) [21:44:41] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:44:48] (03PS1) 10Hashar: docker: php5.5 images [integration/config] - 10https://gerrit.wikimedia.org/r/397634 [21:45:38] PROBLEM - Puppet errors on integration-slave-docker-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:45:42] (03CR) 10Hashar: "I have not really tested those images, that is to eventually migrate the php 5.5 jobs to them :]" [integration/config] - 10https://gerrit.wikimedia.org/r/397634 (owner: 10Hashar) [21:54:42] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:10] (03CR) 10Hashar: "I have pushed the images. Guess I should craft a few jobs now :)" [integration/config] - 10https://gerrit.wikimedia.org/r/397634 (owner: 10Hashar) [22:04:51] 10Release-Engineering-Team, 10Deployments, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3829544 (10greg) Thanks @demon. @Ladsgroup take the reins? [22:06:44] 10Release-Engineering-Team (Kanban), 10User-greg: Explain to TechComm (Daniel K) part of learnings from ORES post-mortem re arch reviews - https://phabricator.wikimedia.org/T182635#3829546 (10greg) p:05Triage>03High [22:07:08] 10Release-Engineering-Team (Kanban), 10User-greg: Explain to TechComm (Daniel K) part of learnings from ORES post-mortem re arch reviews - https://phabricator.wikimedia.org/T182635#3829559 (10greg) https://etherpad.wikimedia.org/p/Post-Mortem-T181006 https://wikitech.wikimedia.org/wiki/Incident_documentation/2... [22:08:09] 10Release-Engineering-Team (Kanban), 10Deployments, 10HHVM, 10Wikimedia-Incident: Figure out why HHVM kept running stale code for hours - https://phabricator.wikimedia.org/T181833#3829563 (10greg) a:03thcipriani @thcipriani will take a stab at some diagnosis here and report back. [22:18:51] 10Continuous-Integration-Infrastructure, 10Readers-Web-Backlog, 10Browser-Tests: Popups browser tests failing as new summary endpoint returns 500 on Main Page - https://phabricator.wikimedia.org/T182465#3829573 (10Jdlrobson) [22:19:01] 10Continuous-Integration-Infrastructure, 10Readers-Web-Backlog, 10Browser-Tests: Popups browser tests failing as new summary endpoint returns 500 on Main Page - https://phabricator.wikimedia.org/T182465#3824042 (10Jdlrobson) [22:20:38] RECOVERY - Puppet errors on integration-slave-docker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [22:24:53] 10Continuous-Integration-Infrastructure, 10Readers-Web-Backlog, 10Browser-Tests: Popups browser tests failing as new summary endpoint returns 500 on Main Page - https://phabricator.wikimedia.org/T182465#3829589 (10Jdlrobson) (note I debuged this by first verifying via https://gerrit.wikimedia.org/r/#/c/39763... [22:27:21] Yippee, build fixed! [22:27:21] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #238: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/238/ [22:37:15] Yippee, build fixed! [22:37:16] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #238: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/238/ [23:18:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Release Pipeline: On CI, upgrade docker-ce from 17.06.2 to 17.09.1 - https://phabricator.wikimedia.org/T177499#3829735 (10hashar) [23:18:57] !log deployed mobileapps@e290b17 to BC [23:19:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:19:12] twentyafterfour: any chance you could CR https://gerrit.wikimedia.org/r/#/c/392221/ ? Are there docs for testing this kind of stuff on deployment-prep (e.g. cherry-pick, puppet master runs, ect)? [23:20:30] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3829740 (10hashar) a:03hashar [23:25:55] AaronSchulz: looking... [23:27:26] 10Continuous-Integration-Config: Puppet CI should fail over CRLF line endings (sometimes) - https://phabricator.wikimedia.org/T182641#3829742 (10EddieGP) [23:29:31] AaronSchulz: I'm not sure about docs but it's fairly straightforward to cherry-pick it onto deployment-puppetmaster in /var/lib/git/operations/puppet [23:29:51] as for puppet runs you can use cumin or just wait for the scheduled runs [23:30:27] ok [23:34:27] AaronSchulz: +1'd [23:35:30] thanks [23:56:07] 10Continuous-Integration-Infrastructure, 10Readers-Web-Backlog, 10Browser-Tests: Popups browser tests failing as new summary endpoint returns 500 on Main Page - https://phabricator.wikimedia.org/T182465#3829774 (10Jdlrobson)