[00:12:11] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3838959 (10EddieGP) [01:02:02] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3838959 (10Dzahn) also: T97593 "Fix beta cluster SSL certs issues" (private ticket but maybe it can be made public, dont know) T50501 "beta: Get SSL certificates for *.{... [01:02:46] 10Beta-Cluster-Infrastructure, 10Cloud-Services, 10Cloud-VPS: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10Dzahn) T182927 has been opened [01:03:58] 10Beta-Cluster-Infrastructure, 10Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#1526857 (10Dzahn) Can this ticket be made public nowadays? There is now T182927 and it seemed like a duplicate. [01:20:38] 10Gerrit, 10User-Zppix: Labs/Icinga2 group ownership - https://phabricator.wikimedia.org/T182941#3839256 (10Zppix) [01:20:52] 10Gerrit, 10VPS-project-icinga2, 10User-Zppix: Labs/Icinga2 group ownership - https://phabricator.wikimedia.org/T182941#3839267 (10Zppix) [01:52:29] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [01:53:21] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:54:48] 10MediaWiki-Codesniffer: ClassMatchesFilename.WrongCase should ignore maintenance scripts - https://phabricator.wikimedia.org/T178725#3700808 (10Legoktm) Not all maintenance scripts are in a sub-directory. I think looking for `RUN_MAINTENANCE_IF_MAIN` is unique and unlikely to have any false positives. [02:43:30] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:49:20] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:11:24] Project mediawiki-core-code-coverage build #3194: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3194/ [03:12:58] oh no [03:13:08] 03:11:20 ...............R.Trying to @cover or @use not existing method "RevisionSlots::getSlot". [03:20:37] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:50:09] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:59:27] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:20:11] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:22:27] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:52:12] Yippee, build fixed! [05:52:13] Project mediawiki-core-code-coverage build #3195: 09FIXED in 1 hr 29 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3195/ [06:55:37] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:51:04] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3838959 (10Krenair) m.$project? I'm not sure those are useful domains? Which is probably why I didn't put them on the list to get certs for [09:33:07] 10Beta-Cluster-Infrastructure, 10Language-2017-Oct-Dec: [betacluster] Special:Translate does not allow to translate article content - multiple "Uncaught SyntaxError: Unexpected token < in JSON at position 0" - https://phabricator.wikimedia.org/T180841#3839731 (10Nikerabbit) [10:49:29] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3839964 (10Aklapper) [12:38:43] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:51:18] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3840295 (10zeljkofilipin) I have just pushed the first test to T182160 br... [13:18:44] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [13:24:12] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840343 (10zeljkofilipin) [13:26:45] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840359 (10zeljkofilipin) [13:26:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840377 (10zeljkofilipin) p:05Triage>03Low [13:29:12] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840381 (10zeljkofilipin) [13:29:14] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3840380 (10zeljkofilipin) 05Open>03stalled [13:29:33] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840383 (10zeljkofilipin) [13:29:35] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:29:37] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3139916 (10zeljkofilipin) [13:32:19] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3840389 (10zeljkofilipin) p:05Normal>03Low [13:32:36] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840392 (10zeljkofilipin) [13:32:38] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3335241 (10zeljkofilipin) [13:38:12] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840395 (10zeljkofilipin) [13:38:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840396 (10zeljkofilipin) [13:38:16] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3840394 (10zeljkofilipin) [13:39:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:39:05] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Create mediawiki-core-qunit-selenium-composer-jessie - https://phabricator.wikimedia.org/T180482#3840397 (10zeljkofilipin) [13:40:00] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:40:02] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3840400 (10zeljkofilipin) [13:40:45] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840403 (10zeljkofilipin) [13:40:47] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125#3840402 (10zeljkofilipin) [13:41:12] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:41:14] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3840404 (10zeljkofilipin) [13:42:57] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3840408 (10zeljkofilipin) [13:42:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) [13:43:01] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125#3840409 (10zeljkofilipin) [13:43:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3271010 (10zeljkofilipin) [13:43:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3271248 (10zeljkofilipin) [13:43:14] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3840411 (10zeljkofilipin) [13:43:18] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Create mediawiki-core-qunit-selenium-composer-jessie - https://phabricator.wikimedia.org/T180482#3840413 (10zeljkofilipin) [13:43:21] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3335098 (10zeljkofilipin) [13:43:33] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3840415 (10EddieGP) >>! In T182927#3839620, @Krenair wrote: > m.$project? I'm not sure those are useful domains? Well they do the same thing they do in prod, redirecting t... [13:45:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840417 (10zeljkofilipin) [13:45:57] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3840418 (10zeljkofilipin) [13:46:00] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Cucumber+Selenium+Node.js in CI - https://phabricator.wikimedia.org/T179190#3840416 (10zeljkofilipin) [13:46:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Cucumber+Selenium+Node.js in CI - https://phabricator.wikimedia.org/T179190#3716147 (10zeljkofilipin) p:05Normal>03Low [13:48:47] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840423 (10zeljkofilipin) [13:48:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840424 (10zeljkofilipin) [13:48:52] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3840422 (10zeljkofilipin) [13:49:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:49:53] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3336361 (10zeljkofilipin) [13:49:55] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3840425 (10zeljkofilipin) [13:50:07] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:50:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3341390 (10zeljkofilipin) [13:50:28] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [13:50:30] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3341396 (10zeljkofilipin) [13:50:33] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Create mediawiki-core-qunit-selenium-composer-jessie - https://phabricator.wikimedia.org/T180482#3840431 (10zeljkofilipin) [14:22:37] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3840487 (10Aklapper) Thanks a lot! I've created GCI tasks for all those items above which say `no mediawiki/minus-x`. (Not all of those GCI ta... [14:43:34] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3840554 (10zeljkofilipin) p:05Normal>03High [14:44:46] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) a:05zeljkofilipin>03None [14:45:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840359 (10zeljkofilipin) a:05zeljkofilipin>03None [15:57:14] 10Continuous-Integration-Config, 10Proton, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3840738 (10hashar) Most of the CI job that uses npm are now on Docker. I will rebuild the i... [16:07:28] (03PS1) 10Hashar: Chromium 63 and set PUPPETEER_SKIP_CHROMIUM_DOWNLOAD [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) [17:25:44] 10Scap, 10Operations, 10Packaging: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3841015 (10akosiaris) After some investigation it turns out I built `master` and not `release` branch which caused all this nice things. I am building `3.7.4-2` to fix this [18:31:53] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3841287 (10debt) [18:31:55] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3841285 (10debt) 0... [18:48:36] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3841335 (10akosiaris) I 've went ahead and built and uploaded the package in order to avoid any immediate issues from the botched release. In D916 jenkins doesn't l... [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:58:14] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3841340 (10rafidaslam) [19:06:12] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:13:55] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, and 2 others: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3841366 (10thcipriani) [19:13:58] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3841364 (10thcipriani) 05Open>03Resolved >>! In T182347#3841335, @akosiaris wrote: > I 've went ahead and built and uploaded the package in order to avoid any i... [19:38:55] (03PS2) 10Hashar: npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) [19:49:10] (03PS3) 10Hashar: npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) [19:50:32] (03PS4) 10Hashar: npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) [19:51:50] (03CR) 10jerkins-bot: [V: 04-1] npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) (owner: 10Hashar) [19:52:41] (03CR) 10Hashar: [C: 032] "That creates the job https://integration.wikimedia.org/ci/job/chromium-render-npm-browser-node-6-docker/ which would be triggerable by co" [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) (owner: 10Hashar) [19:54:01] (03CR) 10jerkins-bot: [V: 04-1] npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) (owner: 10Hashar) [19:55:17] (03PS5) 10Hashar: npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) [19:57:10] (03CR) 10Hashar: [C: 032] npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) (owner: 10Hashar) [19:58:31] (03Merged) 10jenkins-bot: npm job for chromium-render [integration/config] - 10https://gerrit.wikimedia.org/r/398497 (https://phabricator.wikimedia.org/T179552) (owner: 10Hashar) [20:20:54] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:44:40] 10Continuous-Integration-Config, 10Proton, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3841691 (10bmansurov) Pasting what hashar mentioned on IRC (seems like a useful information... [20:46:30] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:49:10] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:53:49] 10Continuous-Integration-Config, 10Proton, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3841696 (10hashar) When using Chromium 62 from Debian, the mocha test suite apparently neve... [20:55:14] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:56:10] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:47:45] 10Scap, 10Operations: scap is broken bash: /srv/deployment/scap/scap/bin/scap: No such file or directory - https://phabricator.wikimedia.org/T183046#3841983 (10Legoktm) p:05Triage>03Unbreak! [23:43:42] 10Scap, 10Operations: scap is broken bash: /srv/deployment/scap/scap/bin/scap: No such file or directory - https://phabricator.wikimedia.org/T183046#3842035 (10thcipriani) [23:46:50] 10Scap, 10Operations: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3842040 (10thcipriani)