[00:11:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#4067254 (10Jdforrester-WMF) [00:11:13] 10RelEng-Archive-FY201718-Q2, 10Release-Engineering-Team, 10Operations, 10WorkType-NewFunctionality: [Spike] Try out hack ( PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:38:14] hey! could someone please enable compiler03.puppet3-diffs.eqiad.wmflabs and disable compiler02.puppet3-diffs.eqiad.wmflabs in jenkins? [00:38:34] herron: Can you not do it yourself? [00:38:36] https://integration.wikimedia.org/ci/ [00:38:40] login with wikitech creds [00:39:02] (if not, I can do it) [00:39:18] hmm, is there an admin area I should see? [00:39:23] Not really [00:39:30] Just click on the host link, ala https://integration.wikimedia.org/ci/computer/compiler02.puppet3-diffs.eqiad.wmflabs/ [00:39:37] Then mark it offline, I guess for now [00:39:48] then on https://integration.wikimedia.org/ci/computer/compiler03.puppet3-diffs.eqiad.wmflabs/ bring it online? :) [00:41:09] should I be seeing something to the effect of online/offline on the left? [00:41:32] top right, dark blue ish button [00:41:40] "Mark this node temporarily offline" [00:42:17] Oh, sorry herron, I'm talking crap [00:42:23] I thought ops all had access [00:42:24] https://github.com/wikimedia/puppet/blob/e3873e1f08d1b6171940657429b66a3aed35c6cb/modules/admin/data/data.yaml#L150-L152 [00:42:41] haha [00:42:48] I mean I am due for some new glasses [00:42:55] or, at least, I think that's what controls the gui stuff [00:42:57] Let me do it [00:43:03] alright, thanks! [00:43:31] !log enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins [00:43:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:44:15] And there's a job running on it... [00:44:22] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/10547/ [00:44:56] nice, that’s me testing it [01:01:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [01:06:49] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [01:38:59] 10Release-Engineering-Team, 10Operations, 10Puppet: puppetdb4: use postgres db backend in puppet-compiler - https://phabricator.wikimedia.org/T187258#4067367 (10herron) compiler03 has been enabled and compiler02 disabled (in jenkins) Running a test puppet compiler build works and looks good. Will proceed w... [02:22:20] (03CR) 10Samwilson: [C: 031] Add QUnit to CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/420833 (https://phabricator.wikimedia.org/T169592) (owner: 10MusikAnimal) [03:07:19] PROBLEM - Puppet errors on deployment-maps01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:27:17] RECOVERY - Puppet errors on deployment-maps01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:27:27] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#4067422 (10Krinkle) 05Resolved>03Open Reverted (again) as of {48b9d506a9... [03:32:36] (03PS3) 10Legoktm: Check for use statement with non-compound name [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420063 (https://phabricator.wikimedia.org/T189323) (owner: 10Umherirrender) [03:32:39] (03CR) 10Legoktm: [C: 032] Check for use statement with non-compound name [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420063 (https://phabricator.wikimedia.org/T189323) (owner: 10Umherirrender) [03:33:18] PROBLEM - Puppet errors on deployment-maps01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:33:37] (03Merged) 10jenkins-bot: Check for use statement with non-compound name [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420063 (https://phabricator.wikimedia.org/T189323) (owner: 10Umherirrender) [03:34:02] (03CR) 10jenkins-bot: Check for use statement with non-compound name [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420063 (https://phabricator.wikimedia.org/T189323) (owner: 10Umherirrender) [03:38:05] (03PS2) 10Legoktm: Don't require @covers from abstract tests [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420259 (https://phabricator.wikimedia.org/T187738) (owner: 10MaxSem) [03:38:10] (03CR) 10Legoktm: [C: 032] Don't require @covers from abstract tests [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420259 (https://phabricator.wikimedia.org/T187738) (owner: 10MaxSem) [03:38:27] 10MediaWiki-Codesniffer, 10Patch-For-Review: UnusedUsed sniff did not catch use from global namespace - https://phabricator.wikimedia.org/T189323#4067439 (10Legoktm) 05Open>03Resolved [03:38:57] (03Merged) 10jenkins-bot: Don't require @covers from abstract tests [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420259 (https://phabricator.wikimedia.org/T187738) (owner: 10MaxSem) [03:40:01] (03CR) 10jenkins-bot: Don't require @covers from abstract tests [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420259 (https://phabricator.wikimedia.org/T187738) (owner: 10MaxSem) [03:40:59] (03CR) 10Legoktm: [C: 04-1] Prohibit nested functions (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) (owner: 10MaxSem) [03:43:20] RECOVERY - Puppet errors on deployment-maps01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:48:58] (03CR) 10Legoktm: [C: 032] Add totals to deploy-notes [tools/release] - 10https://gerrit.wikimedia.org/r/416873 (owner: 10Thcipriani) [03:49:28] (03Merged) 10jenkins-bot: Add totals to deploy-notes [tools/release] - 10https://gerrit.wikimedia.org/r/416873 (owner: 10Thcipriani) [03:50:12] 10MediaWiki-Codesniffer, 10Patch-For-Review: MissingCovers sniff should not warn on abstract classes - https://phabricator.wikimedia.org/T187738#4067444 (10Legoktm) 05Open>03Resolved a:03MaxSem [05:43:57] (03CR) 10MaxSem: Prohibit nested functions (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) (owner: 10MaxSem) [06:07:07] (03CR) 10Legoktm: [C: 04-1] "Thiemo is right, it should be rather straightforward to handle this as a separate sniff." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420159 (https://phabricator.wikimedia.org/T182057) (owner: 10MaxSem) [06:49:28] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4061790 (10Nikerabbit) Also seen in https://gerrit.wikimedia.org/r/c/420929/ for ContentTranslation. I can't see the errors lo... [06:49:45] there is something wrong with my phone or my Google account. [07:30:50] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4067609 (10Nikerabbit) So I assume some dependency that is not specified with exact version got updated and broke things. Dif... [08:02:26] 10Continuous-Integration-Infrastructure (shipyard): Investigate usage of Docker volume for CI - https://phabricator.wikimedia.org/T179742#4067633 (10hashar) 05Open>03declined Will do that with Kubernetes if and when we move to it. [08:04:34] 10Continuous-Integration-Infrastructure (shipyard), 10Quibble: Start mysql in quibble container - https://phabricator.wikimedia.org/T166145#4067654 (10hashar) One has to create a directory then populate the basics: mysql_install_db --datadir=/tmp/foo --user="$USER" To boot it as current user without TCP bu... [08:07:57] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Switch CI Docker Storage Driver to its own partition and to use devicemapper - https://phabricator.wikimedia.org/T178663#4067655 (10hashar) [09:16:39] (03PS6) 10Hashar: Add support for sqlite/mysql databases [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (https://phabricator.wikimedia.org/T166145) [09:17:02] (03CR) 10jerkins-bot: [V: 04-1] Add support for sqlite/mysql databases [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (https://phabricator.wikimedia.org/T166145) (owner: 10Hashar) [09:24:14] (03PS7) 10Hashar: Add support for sqlite/mysql databases [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (https://phabricator.wikimedia.org/T166145) [09:24:16] (03PS1) 10Hashar: Tweak flake8 excludes [integration/quibble] - 10https://gerrit.wikimedia.org/r/420971 [09:25:52] (03CR) 10Hashar: [C: 032] Tweak flake8 excludes [integration/quibble] - 10https://gerrit.wikimedia.org/r/420971 (owner: 10Hashar) [09:26:16] (03Merged) 10jenkins-bot: Tweak flake8 excludes [integration/quibble] - 10https://gerrit.wikimedia.org/r/420971 (owner: 10Hashar) [09:40:37] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4067819 (10Nikerabbit) FWIW: On my Ubuntu server (over ssh) with node v7.0.0 and npm 3.10.8 I don't observe the "dots" issue,... [10:12:22] (03CR) 10Hashar: [C: 032] Add QUnit to CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/420833 (https://phabricator.wikimedia.org/T169592) (owner: 10MusikAnimal) [10:13:38] (03Merged) 10jenkins-bot: Add QUnit to CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/420833 (https://phabricator.wikimedia.org/T169592) (owner: 10MusikAnimal) [10:45:53] Hi, do we have a github bot that can link pull requests back to phab task, like we do for gerrit? [10:52:23] No. [10:52:37] stephanebisson: for GerritBot, see https://wikitech.wikimedia.org/wiki/Gerrit_Notification_Bot [10:52:49] if you want to write and run a bot on GitHub you are free to. :P [10:53:42] #wikimedia-devtools might be more suited for the topic [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:54] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:14:50] Project mwext-phpunit-coverage-publish build #2427: 04FAILURE in 3.3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2427/ [11:21:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 2 others: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4068168 (10zeljkofilipin) a... [11:26:13] Yippee, build fixed! [11:26:14] Project mwext-phpunit-coverage-publish build #2428: 09FIXED in 1 min 56 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2428/ [11:33:40] Project mwext-phpunit-coverage-publish build #2430: 04FAILURE in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2430/ [11:36:27] Yippee, build fixed! [11:36:28] Project mwext-phpunit-coverage-publish build #2431: 09FIXED in 2 min 39 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2431/ [11:42:09] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068266 (10zeljkofilipin) [11:42:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068277 (10zeljkofilipin) p:05Triage>03Unbreak! [11:45:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068285 (10zeljkofilipin) Comparing two builds (good [[ https://integration.wikimedia.org/ci/job/mediawiki-core-npm-node-6-docker/1129/console | 1129 ]], bad [[ https://in... [11:46:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068288 (10zeljkofilipin) I was able to reproduce the problem locally: ``` rm -rf node_modules/ npm i npm t ``` [11:49:07] (03Draft1) 10MarcoAurelio: Archive the Annotator extension [integration/config] - 10https://gerrit.wikimedia.org/r/420995 (https://phabricator.wikimedia.org/T189753) [11:49:11] (03PS2) 10MarcoAurelio: Archive the Annotator extension [integration/config] - 10https://gerrit.wikimedia.org/r/420995 (https://phabricator.wikimedia.org/T189753) [12:07:27] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068375 (10zeljkofilipin) ``` $ npm ls postcss-less /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki └─┬ stylelint@8.2.0 └── postcss-less@1.1.4 ``` [12:09:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Warning: Task "stylelint:src" failed - https://phabricator.wikimedia.org/T190269#4068393 (10zeljkofilipin) [[ https://github.com/stylelint/stylelint/blob/master/package.json#L65 | stylelint/package.json ]] ``` ... "postcss-less": "^1.1.0", ... ``` [12:23:37] 10Phabricator: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068443 (10MarcoAurelio) [12:27:08] 10Phabricator: acl*phabricator description update - https://phabricator.wikimedia.org/T174279#4068458 (10MarcoAurelio) Maybe adding `#acl_releng` and `#opsacl` to the 4th bullet description would look prettier too (without the commas to have them linked). Thanks. [12:27:37] 10Phabricator (Upstream), 10Upstream: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068459 (10Aklapper) p:05Triage>03Lowest Unlikely to happen I'd say. [12:27:55] (03CR) 10Hashar: [C: 032] Archive the Annotator extension [integration/config] - 10https://gerrit.wikimedia.org/r/420995 (https://phabricator.wikimedia.org/T189753) (owner: 10MarcoAurelio) [12:29:09] (03Merged) 10jenkins-bot: Archive the Annotator extension [integration/config] - 10https://gerrit.wikimedia.org/r/420995 (https://phabricator.wikimedia.org/T189753) (owner: 10MarcoAurelio) [12:32:46] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4068471 (10hashar) [12:33:45] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4068474 (10hashar) >>! In T190072#4067609, @Nikerabbit wrote: > > ``` > -│ ├─┬ postcss-less@1.1.3 > +│ ├─┬ postcss-less... [12:38:48] (03Draft1) 10MarcoAurelio: Mark repository as read-only [extensions/Annotator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421004 [12:38:50] (03PS2) 10MarcoAurelio: Mark repository as read-only [extensions/Annotator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421004 (https://phabricator.wikimedia.org/T189753) [12:41:42] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4068496 (10zeljkofilipin) [12:43:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) [12:43:46] (03CR) 10MarcoAurelio: [V: 032 C: 032] Mark repository as read-only [extensions/Annotator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421004 (https://phabricator.wikimedia.org/T189753) (owner: 10MarcoAurelio) [12:43:59] (03CR) 10MarcoAurelio: [V: 032 C: 032] "to submit later" [extensions/Annotator] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421004 (https://phabricator.wikimedia.org/T189753) (owner: 10MarcoAurelio) [12:45:30] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4068499 (10zeljkofilipin) [12:46:05] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) [12:58:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests: contint-admins sudo for service jenkins - https://phabricator.wikimedia.org/T190277#4068588 (10hashar) [13:34:47] 10Phabricator (Upstream), 10Upstream: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068681 (10Aklapper) The UI allows admins to remove comments in tasks. There is no such dropdown in Conpherence. In the CLI, `sudo ./phabricator/bin/remove destroy` requires an ID to pass... [13:36:32] 10Phabricator (Upstream), 10Upstream: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068683 (10MarcoAurelio) I'd say that, at least, something should be implemented via CLI or web. I think it's a risk that, ie, outing can remain indefinitely in a room; unless maybe delet... [13:37:08] 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster Obama page often responds with 503 - https://phabricator.wikimedia.org/T188913#4068684 (10Niedzielski) It's consistently text04: ``` Request from 73.252.38.252 via deployment-cache-text04 deployment-cache-text04, Varnish XID 161469920 Error: 503, Backe... [13:51:26] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4068710 (10zeljkofilipin) [13:53:12] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) [14:25:54] (03Draft1) 10MarcoAurelio: Archive PushToWatch [integration/config] - 10https://gerrit.wikimedia.org/r/421021 (https://phabricator.wikimedia.org/T190286) [14:26:07] (03PS2) 10MarcoAurelio: Archive PushToWatch [integration/config] - 10https://gerrit.wikimedia.org/r/421021 (https://phabricator.wikimedia.org/T190286) [14:36:05] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4068906 (10zeljkofilipin) 05Open>03Resolved [14:36:58] Project mwext-phpunit-coverage-publish build #2437: 04FAILURE in 3.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2437/ [14:38:23] ERROR: Step ‘Archive the artifacts’ failed: no workspace for mwext-phpunit-coverage-publish #2437 [14:38:51] 10Phabricator (Upstream), 10Upstream: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068917 (10Aklapper) [[ https://secure.phabricator.com/maniphest/query/aCCmAfy7WdqT/#R | Upstream tasks about Conpherence ]] (I don't see a related one); https://discourse.phabricator-com... [14:44:36] 10Phabricator (Upstream), 10Upstream: Allow to delete messages from Conpherence - https://phabricator.wikimedia.org/T190276#4068949 (10Aklapper) Also relevant is probably if you have experienced problematic cases in reality and if so which type ("is this a theoretical problem or an existing one?"). [14:45:45] Yippee, build fixed! [14:45:46] Project mwext-phpunit-coverage-publish build #2438: 09FIXED in 2 min 20 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2438/ [14:50:57] (03CR) 10Hashar: [C: 032] Archive PushToWatch [integration/config] - 10https://gerrit.wikimedia.org/r/421021 (https://phabricator.wikimedia.org/T190286) (owner: 10MarcoAurelio) [14:52:10] (03Merged) 10jenkins-bot: Archive PushToWatch [integration/config] - 10https://gerrit.wikimedia.org/r/421021 (https://phabricator.wikimedia.org/T190286) (owner: 10MarcoAurelio) [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:53:19] no_justification: can I be temporarily added to https://gerrit.wikimedia.org/r/#/admin/groups/532,members so I can finish archiving the extension? [15:02:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4069059 (10zeljkofilipin) [15:03:24] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) [15:31:16] PROBLEM - Puppet staleness on deployment-maps01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [15:32:07] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4069147 (10zeljkofilipin) [15:32:53] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4069150 (10zeljkofilipin) [15:32:58] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade WebdriverIO to 4.12.0 - https://phabricator.wikimedia.org/T180144#4069149 (10zeljkofilipin) 05Open>03Resolved [15:35:49] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:53:05] 10Project-Admins, 10Community-Liaisons (Jan-Mar-2018): Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069234 (10Elitre) [15:53:52] 10Project-Admins, 10Community-Liaisons (Jan-Mar-2018): Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069247 (10Aklapper) Any project description? [15:56:11] 10Project-Admins, 10Community-Liaisons (Jan-Mar-2018): Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069267 (10Elitre) "Tag for changes that require tech ambassadors' attention and/or action. A proposed working tool for the Tech Ambassadors group: https://meta.wikime... [15:59:02] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4069292 (10greg) [16:05:01] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4069341 (10greg) [16:06:32] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4017888 (10greg) Meeting happened on Tuesday Mar 20th. Etherpad: https://etherpad.wikimedia.org/p/20180320-SeleniumRetrospective [16:06:40] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069234 (10Elitre) [16:06:57] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069234 (10Elitre) [16:10:21] PROBLEM - Host integration-slave-k8s-1010 is DOWN: CRITICAL - Host Unreachable (10.68.19.44) [16:10:51] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:16] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4069382 (10zeljkofilipin) [16:22:18] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Use createAccount() in Selenium tests - https://phabricator.wikimedia.org/T180379#4069378 (10zeljkofilipin) 05Open>03declined Because of {T181284}. [16:24:48] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Investigate if WebdriverIO `sync: false` would be useful to us and document how to use it - https://phabricator.wikimedia.org/T182412#4069403 (10zeljkofilipin) [16:47:29] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4069499 (10zeljkofilipin) [16:47:48] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4069514 (10zeljkofilipin) p:05Triage>03High [16:50:50] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<100.00%) [16:51:56] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4069523 (10zeljkofilipin) [16:52:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4069525 (10zeljkofilipin) [16:52:30] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4069499 (10zeljkofilipin) [16:55:05] 10Release-Engineering-Team (Kanban), 10Vector, 10MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)), 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#4069533 (10zeljkofilipin) [16:55:17] 10Release-Engineering-Team (Kanban), 10Vector, 10MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)), 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#3988395 (10zeljkofilipin) a:05zeljko... [16:58:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: contint-admins sudo for service jenkins - https://phabricator.wikimedia.org/T190277#4069554 (10RobH) As @MoritzMuehlenhoff pointed out on the patch comment h... [16:58:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4069561 (10zeljkofilipin) [17:02:35] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4069582 (10zeljkofilipin) To make it explicit: I think this is ready. I am just waiting for a review or two to let... [17:03:22] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, 10Performance-Team (Radar): Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#4069587 (10hashar) [17:04:37] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, 10Performance-Team (Radar): Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#3543271 (10hashar) I guess that is the same issue we had on production. Pro... [17:04:41] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4069597 (10zeljkofilipin) [17:20:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#3785137 (10zeljkofilipin) [17:26:57] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4069731 (10zeljkofilipin) [17:28:42] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#3785137 (10zeljkofilipin) [17:52:00] !log deployment-prep: scap sync-file docroot/wwwportal/portal 'T173887' [17:52:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:52:02] T173887: Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887 [17:53:43] Project beta-scap-eqiad build #200418: 04FAILURE in 0.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/200418/ [17:56:30] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, 10Performance-Team (Radar): Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#4069922 (10hashar) I did a sync of docroot/wwwportal/portal but that hasn't... [18:06:21] Yippee, build fixed! [18:06:21] Project beta-scap-eqiad build #200419: 09FIXED in 5 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/200419/ [18:12:39] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, 10Performance-Team (Radar): Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#4069980 (10hashar) At least https://www.wikipedia.beta.wmflabs.org/apple-app... [18:18:57] !log fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there [18:18:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:12] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:19:40] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:29:25] but before doing that we'd need to migrate varnishkafka in deployment-prep to jumbo (if not already done?) [18:31:38] kafka.metadata.broker.list = deployment-kafka-jumbo-1.deployment-prep.eqiad.wmflabs:9093,deployment-kafka-jumbo-2.deployment-prep.eqiad.wmflabs:9093 [18:33:04] so now I am a bit confused, how was the kafka_config() working before now? [18:33:48] ahhh the joy of labs [18:34:31] and of course eventlogging was already configured for jumbo in labs [18:34:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.008 second response time [18:35:27] hey releng.. is jenkins having issues today? ping hashar [18:35:43] seeing all sorts of issues trying to merge https://gerrit.wikimedia.org/r/#/c/419812/ [18:35:59] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:36:06] Project mwext-phpunit-coverage-publish build #2444: 04FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2444/ [18:36:50] 18:33:10 wikimedia/purtle: 1.0.7 installed, 1.0.6 required. [18:36:50] 18:33:10 Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies [18:37:27] jdlrobson: hi! Was it working before and now it doesn't ? [18:37:36] somebody updated a dep, but forgot to update it in mw core [18:37:37] (before meaning an hour ago or earlier) [18:37:46] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4070077 (10Lea_WMDE) @Tonina_Zhelyazkova_WMDE @thiemowmde @aniansson could you check the readme, please? [18:38:03] ah okok I thought that unblocking puppet sync caused side effects [18:38:07] seems it was 21 hours ago [18:38:08] https://github.com/wikimedia/mediawiki-vendor/commit/f6c0f8f4c3b13d981949fbcf54189cc1a87a8d68 [18:38:25] SMalyshev ^^ [18:38:58] paladox: yes? [18:39:14] SMalyshev you need to update purtle in mw core too [18:39:20] :) [18:39:36] SMalyshev see https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/39933/console [18:39:37] paladox: elukey not sure the cause but getting lots of strange jenkins failures on different jobs unrelated to changes [18:39:46] jdlrobson the cause is: [18:39:49] wasn't an issue yesterday but is today :) [18:39:51] https://github.com/wikimedia/mediawiki-vendor/commit/f6c0f8f4c3b13d981949fbcf54189cc1a87a8d68 [18:39:54] jdlrobson ^^ [18:40:01] that needs updating in mw core too [18:40:12] (has to be done in both mw vendor and mw core at the same time) [18:40:30] is someone fixing that? anything i can help with? [18:40:39] !log Re-added ArielGlenn as deployment-prep project admin [18:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:40:42] paladox: ok, made https://gerrit.wikimedia.org/r/c/421101/ [18:40:54] thanks [18:40:56] jdlrobson ^^ [18:41:00] I wonder what's the right way to do it then? looks like it needs 2 patches [18:41:10] and they cross-depend on each other [18:41:15] yep [18:43:03] I wonder if we shouldn't use ^1.0.6 or something in core... the last word is vendor repo anyway... [18:45:53] jdlrobson wondering if you could merge https://gerrit.wikimedia.org/r/#/c/421101/ please? [18:46:01] (to unbreak all the tests) [18:46:14] paladox: can you add a bug to the commit msg? [18:46:17] to provide context [18:46:28] jdlrobson there is no bug. [18:46:41] should there be? [18:46:48] Or an associated commit? [18:46:49] I can add the bug for 1.0.7 upgrade if you want [18:47:00] SMalyshev: thanks that would be great [18:47:00] jdlrobson depends, this happends with oo** sometimes [18:48:03] done [18:48:34] jdlrobson ^^ [18:48:54] +2ed [18:48:58] thanks for the context! [18:49:17] sorry for the mess, I forgot that it should be updated in two places :( [18:49:19] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4070093 (10zeljkofilipin) I do not know if [[ https://phabricator.wikimedia.org/diffusion/EASR/browse/master/tests/selenium/README.md;4ec8774228815... [18:51:13] https://integration.wikimedia.org/ci/job/mediawiki-core-php70-phan-docker/7052/console [18:51:14] fails [18:51:15] with [18:51:23] 18:49:37 git.exc.GitCommandError: 'git clone -v /srv/git/mediawiki/core.git /src' returned with exit code 128 [18:51:24] 18:49:37 stderr: 'fatal: destination path '/src' already exists and is not an empty directory. [18:51:27] hashar ^^ [18:52:50] hmm what's that? [18:52:58] phan [18:53:07] * paladox dosen't know what phan does though [18:53:12] or legoktm ^^ [18:53:15] phan is a syntax checker [18:53:19] static analysis [18:53:20] oh [18:53:37] but no idea why the build fails there... can be temporary issue [18:53:44] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4070105 (10MarcoAurelio) There seems to be a problem with how I implemented this: ``` -----... [18:53:47] SMalyshev seems it is because of [18:53:51] [18:51:23] 18:49:37 stderr: 'fatal: destination path '/src' already exists and is not an empty directory. [18:54:13] yeah looks like some problem with setup... I'd let the jobs finish and try again [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:37] SMalyshev but worked in the test pipeline [18:54:41] but failed in gate and submit [18:54:50] yeah weird agree [18:56:37] hmm code coverage seems to be stalled [18:56:38] https://integration.wikimedia.org/zuul/ [18:56:47] (not related to the mw change, seperate) [18:59:12] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:43] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:53] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, 10Performance-Team (Radar): Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#4070119 (10Jdrewniak) @hashar that sounds good to me! anything that helps s... [19:05:38] (03Draft2) 10MarcoAurelio: Mark repository as read only [extensions/PushToWatch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421106 [19:05:45] (03Draft1) 10MarcoAurelio: Mark repository as read only [extensions/PushToWatch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/421106 [19:14:03] ok, core patch is in an the problems should be gone [19:14:04] paladox: ^^ [19:14:14] SMalyshev thanks :) [19:19:43] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#4070188 (10demon) a:05demon>03None [19:21:17] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4069234 (10Aklapper) 05Open>03Resolved a:03Aklapper Requested public project #tech-ambassadors has been created: https://phabricator.wikimedia.org/project/view/3302/ Please encourage interested peopl... [19:28:21] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Set up puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#4070238 (10Andrew) [19:29:29] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM, and 4 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#4070240 (10Krinkle) [19:35:26] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM, and 4 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#4070287 (10Krinkle) 05Open>03Resolved a:03Krinkle As of a few weeks ago... [19:35:58] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM, and 4 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#4070299 (10Krinkle) [19:37:21] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1165897 (10Krinkle) [19:39:00] 10Deployments, 10MediaWiki-Debug-Logger, 10User-Tgr: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#4070326 (10Krinkle) [20:03:41] 10Phabricator: Develop a custom herald condition for maniphest tasks: "projects include any subprojects of ___ " - https://phabricator.wikimedia.org/T144041#4070424 (10mmodell) I took a shot at implementing this last night and it turns out to be a lot more difficult than it should be. It would require loading o... [20:07:26] * paladox updates polygerrit syntax highlight colours in the diff https://gerrit-review.googlesource.com/c/gerrit/+/163952 :) [20:13:25] paladox: still seems a lot of a backlog :/ ? https://integration.wikimedia.org/zuul/ [20:13:38] waiting on mediawiki-extensions-hhvm-jessie [20:13:41] yep [20:13:51] i guess some tests are still on nodepool [20:19:40] !log deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to e6b50a0 [20:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:21:53] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#4070515 (10Tgr) [20:24:07] Yippee, build fixed! [20:24:07] Project mwext-phpunit-coverage-publish build #2445: 09FIXED in 2 min 14 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2445/ [20:41:36] 10Phabricator: Develop a custom herald condition for maniphest tasks: "projects include any subprojects of ___ " - https://phabricator.wikimedia.org/T144041#4070581 (10MBinder_WMF) Thanks for taking a stab at it! Bummer that it's difficult. To clarify, this is a suggested approach to a core issue of "it's annoyi... [20:48:41] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:53:59] 10Phabricator: should be possible to do a search across all android phab tasks (or iOS or mobilefrontend, etc.) - https://phabricator.wikimedia.org/T190342#4070617 (10jeremyb) [21:07:16] 10Phabricator: should be possible to do a search across all android phab tasks (or iOS or mobilefrontend, etc.) - https://phabricator.wikimedia.org/T190342#4070667 (10Aklapper) 05Open>03stalled The first is a project, the second is a milestone of that project. https://www.mediawiki.org/wiki/Phabricator/Proje... [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:08:51] 10Phabricator: should be possible to do a search across all android phab tasks (or iOS or mobilefrontend, etc.) - https://phabricator.wikimedia.org/T190342#4070678 (10Aklapper) I tested with https://phabricator.wikimedia.org/T186768#4002668 which is included in https://phabricator.wikimedia.org/maniphest/query/T... [21:09:18] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: [Blocked] Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4070684 (10mmodell) @awight: Scap 3.7.7 is deployed now, are you... [21:10:06] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: [Blocked] Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4070691 (10awight) Cool! Yes it was 3.7.7 I was waiting for, wi... [21:10:28] 10Release-Engineering-Team (Watching / External), 10Operations, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4070707 (10awight) [21:10:38] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team (Current), 10Wikimedia-Incident: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix - https://phabricator.wikimedia.org/T181010#4070710 (10awight) [21:10:41] 10Release-Engineering-Team (Watching / External), 10Operations, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3778593 (10awight) 05stalled>03Open [21:11:05] 10Release-Engineering-Team (Watching / External), 10Operations, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4070712 (10mmodell) I probably _should_ have called it 3.8 ;) [21:13:42] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:14:47] greg-g: I'm planning to a late deployment of the ORES service today, do you have an opinion about whether this is a good time to do so? [21:15:01] 10Phabricator: should be possible to do a search across all android phab tasks (or iOS or mobilefrontend, etc.) - https://phabricator.wikimedia.org/T190342#4070731 (10jeremyb) 05stalled>03Open hrmmm, ok, works for maniphest search but not global search? https://phabricator.wikimedia.org/search/query/P2VUrKR... [21:15:28] It's supposed to be a no-op, just moving the supporting virtualenv [21:17:05] rc4 of gerrit 2.15 was released https://groups.google.com/forum/#!topic/repo-discuss/ie8luafzzog :) [21:17:19] 10Phabricator: In Phabricator's global search, tasks in a project's milestone are not found when searching in the parent project - https://phabricator.wikimedia.org/T190342#4070752 (10Aklapper) [21:19:19] 10Phabricator: In Phabricator's global search, tasks in a project's milestone are not found when searching in the parent project - https://phabricator.wikimedia.org/T190342#4070770 (10jeremyb) removing Android folks if they don't want the spam [21:19:22] greg-g: On second thought, I'll do it during a scheduled window, tomorrow. [21:19:43] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:22:23] 10Differential, 10Wikimedia Phabricator RfC, 10Gerrit-Migration: Provide example .arclint a standard set of lints/checks relevant to our community - https://phabricator.wikimedia.org/T170#4070778 (10demon) 05Open>03declined [21:23:11] awight: better idea :) [21:24:11] :) I had an initial rush of "my task is unblocked", before the darn better judgment kicked in. [21:29:03] 10Phabricator (Upstream), 10Upstream: Upon edit, a task description which mentions a Phab user (re)adds that Phab user to CC/Subscribers field - https://phabricator.wikimedia.org/T96464#1217738 (10mmodell) I would think that this is less important now that tasks have an explicit mute which overrides notificati... [21:36:48] no_justification i do like the error screen in pg @ https://gerrit.wikimedia.org/r/c/99101 [21:42:20] 10Release-Engineering-Team (Watching / External), 10Operations, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4070861 (10awight) [21:44:26] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4068266 (10Jdforrester-WMF) Not really resolved; we're g... [21:44:44] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:35] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4070880 (10Jdforrester-WMF) Aha, there's now `postcss-le... [21:49:02] Project mwext-phpunit-coverage-publish build #2451: 04FAILURE in 2 min 13 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2451/ [21:51:38] Yippee, build fixed! [21:51:39] Project mwext-phpunit-coverage-publish build #2452: 09FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2452/ [22:15:18] is there any work done about https://phabricator.wikimedia.org/T189762 ? these tests still fail randomly and it's very annoying [22:16:16] RECOVERY - Puppet staleness on deployment-maps01 is OK: OK: Less than 1.00% above the threshold [3600.0] [22:21:19] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:31] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:14] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:54] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:24:22] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:52] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36422 bytes in 7.398 second response time [22:34:20] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 47503 bytes in 7.082 second response time [22:36:14] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48040 bytes in 5.628 second response time [22:37:26] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 47503 bytes in 4.721 second response time [22:38:10] RECOVERY - App Server Main HTTP Response on deployment-mediawiki05 is OK: HTTP OK: HTTP/1.1 200 OK - 47463 bytes in 5.037 second response time [22:40:07] !log cancel beta-update-database-eqiad build in Jenkins (deadlock) [22:40:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:40:29] !log cancel beta-code-update build in Jenkins (deadlock) [22:40:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:47:56] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4071090 (10hashar) I guess we can revert the workaround... [22:50:11] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4071098 (10Jdforrester-WMF) Not until the caches are pur... [23:04:00] PROBLEM - Host deployment-maps01 is DOWN: PING CRITICAL - Packet loss = 100% [23:16:34] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4071197 (10Jdforrester-WMF) [23:16:39] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4071200 (10Jdforrester-WMF) [23:17:42] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4068266 (10Jdforrester-WMF) 05Resolved>03Open p:05U... [23:25:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4071256 (10mmodell) I fixed the quotes and a couple of other issues with the job so that jenkins bot will actually co... [23:26:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4071257 (10mmodell) p:05Triage>03Normal [23:26:45] 10Phabricator, 10Release-Engineering-Team (Kanban), 10media-storage: Connect Phabricator to swift for storage of git-lfs and file uploads. - https://phabricator.wikimedia.org/T182085#4071258 (10mmodell) p:05Normal>03High [23:43:05] RECOVERY - Host deployment-maps01 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms