[00:01:11] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 should support post-deploy checks - https://phabricator.wikimedia.org/T109515#1707533 (10dduvall) [00:16:19] !log Restored normal mysql replication on deployment-beta [00:16:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:55:40] PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [00:58:52] PROBLEM - Puppet staleness on angry-caching-proxy is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [01:07:49] PROBLEM - Puppet staleness on deployment-cache-parsoid04 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [01:15:00] PROBLEM - Puppet staleness on deployment-conf03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [01:34:45] angry-caching-proxy is angry [02:06:20] 10MediaWiki-Codesniffer: Add rule for unused variables - https://phabricator.wikimedia.org/T114846#1707678 (10Krinkle) 3NEW [02:06:33] 10MediaWiki-Codesniffer: Add rule for unused variables - https://phabricator.wikimedia.org/T114846#1707685 (10Krinkle) [02:38:48] Project beta-scap-eqiad build #73138: 04FAILURE in 4 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/73138/ [02:42:57] 02:38:00 scap-rebuild-cdbs failed: 'bool' object has no attribute 'encode' [02:50:18] 10Deployment-Systems, 6Release-Engineering-Team: Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#1707729 (10Krinkle) Ran `convert-wmf-branches 1.26` on repos `mediawiki/core` and `mediawiki/extensions/{VisualEditor,NavigationTiming}`. [03:11:26] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #741: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/741/ [03:48:40] 10Deployment-Systems: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1707757 (10greg) 3NEW [03:48:58] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1707764 (10greg) [03:50:48] 10MediaWiki-Codesniffer: Add rule for unused variables - https://phabricator.wikimedia.org/T114846#1707767 (10Legoktm) We already have a sniff that is able to detect unused global variables, something like it could be used to detect normal variables that are unused. [04:22:30] Yippee, build fixed! [04:22:31] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #584: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/584/ [05:25:46] Yippee, build fixed! [05:25:47] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #564: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/564/ [05:50:05] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Figure out why PHPCS is unable to parse FormatMetadata.php - https://phabricator.wikimedia.org/T113871#1707875 (10TasneemLo) Legoktm, Could you elaborate on how to reproduce this ? I took the latest `includes/media/FormatMetadata.php` from core and I do... [06:13:35] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Figure out why PHPCS is unable to parse FormatMetadata.php - https://phabricator.wikimedia.org/T113871#1707890 (10Legoktm) Umm, what. I could've sworn this failed, but https://gerrit.wikimedia.org/r/#/c/244119/ is passing now...https://integration.wikime... [06:23:06] (03CR) 10Legoktm: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:29:06] (03CR) 10jenkins-bot: [V: 04-1] Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:36:28] (03PS2) 10TasneemLo: Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) [06:44:33] (03CR) 10Legoktm: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:46:29] (03CR) 10jenkins-bot: [V: 04-1] Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:52:58] (03PS3) 10Legoktm: Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:54:25] (03PS4) 10Legoktm: Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:58:40] (03CR) 10Legoktm: [C: 032] "Looks good, thanks :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:59:00] (03Merged) 10jenkins-bot: Sniff to warn check usage of dirname(__FILE__) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [06:59:22] 10MediaWiki-Codesniffer, 5Patch-For-Review: Add sniff to detect usage of dirname( __FILE__ ) instead of __DIR__ - https://phabricator.wikimedia.org/T114267#1707935 (10Legoktm) 5Open>3Resolved [07:05:35] (03PS7) 10Legoktm: Add IfElseStructureSniff to handle else structures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237733 (https://phabricator.wikimedia.org/T101311) (owner: 10TasneemLo) [07:09:39] (03CR) 10Legoktm: [C: 032] "Nice :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237733 (https://phabricator.wikimedia.org/T101311) (owner: 10TasneemLo) [07:10:05] (03Merged) 10jenkins-bot: Add IfElseStructureSniff to handle else structures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237733 (https://phabricator.wikimedia.org/T101311) (owner: 10TasneemLo) [07:12:39] 10MediaWiki-Codesniffer, 7Easy, 5Patch-For-Review: Add sniff for checking that "else" is on same line as previous closing brace - https://phabricator.wikimedia.org/T101311#1707937 (10Legoktm) 5Open>3Resolved [07:16:30] (03PS3) 10Legoktm: Sniff to check assignment in while & if [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237902 (https://phabricator.wikimedia.org/T92744) (owner: 10TasneemLo) [07:27:02] (03CR) 10Legoktm: [C: 04-1] "The following fails and should not:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237902 (https://phabricator.wikimedia.org/T92744) (owner: 10TasneemLo) [07:30:31] (03CR) 10TasneemLo: "Will look into it" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237902 (https://phabricator.wikimedia.org/T92744) (owner: 10TasneemLo) [07:38:59] (03CR) 10TasneemLo: "There seems to be 2 ways to handle this." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237902 (https://phabricator.wikimedia.org/T92744) (owner: 10TasneemLo) [07:53:54] 10Deployment-Systems, 7Beta-Cluster-reproducible: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1707969 (10hashar) [07:55:39] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 7Beta-Cluster-reproducible: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1707757 (10hashar) [08:11:28] (03PS4) 10TasneemLo: Sniff to check assignment in while & if [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/237902 (https://phabricator.wikimedia.org/T92744) [08:16:29] 10MediaWiki-Codesniffer, 5MW-1.27-release-notes, 7Upstream, 5WMF-deploy-2015-10-06_(1.27.0-wmf.2): @codingStandardsIgnoreStart only works with // comments - https://phabricator.wikimedia.org/T114213#1708009 (10TasneemLo) This seems unusual. I was testing this, and it looks as though `// @codingStandardsIg... [08:18:01] 10MediaWiki-Codesniffer, 5MW-1.27-release-notes, 7Upstream, 5WMF-deploy-2015-10-06_(1.27.0-wmf.2): @codingStandardsIgnoreStart only works with some types of comments - https://phabricator.wikimedia.org/T114213#1708010 (10Amire80) [08:18:20] 10MediaWiki-Codesniffer, 5MW-1.27-release-notes, 7Upstream, 5WMF-deploy-2015-10-06_(1.27.0-wmf.2): @codingStandardsIgnoreStart only works with some types of comments - https://phabricator.wikimedia.org/T114213#1688346 (10Amire80) >>! In T114213#1708009, @TasneemLo wrote: > This seems unusual. I was testing... [08:22:41] zeljkof: example : https://gerrit.wikimedia.org/r/#/c/241671/2/jjb/python-jobs.yaml,unified [08:24:40] zeljkof: and https://gerrit.wikimedia.org/r/#/c/241671/2/zuul/layout.yaml,unified [08:24:56] which injects NODE_OFFLINE_WHEN_COMPLETE=1 to the job so nodepool garbage collect them [08:25:13] 10Continuous-Integration-Config: Move bundler Jenkins jobs to nodepool instances - https://phabricator.wikimedia.org/T114860#1708023 (10zeljkofilipin) 3NEW a:3zeljkofilipin [08:29:40] (03CR) 10Paladox: "@Hashar this patch https://gerrit.wikimedia.org/r/#/c/243379/ was merged." [integration/config] - 10https://gerrit.wikimedia.org/r/243380 (owner: 10Paladox) [08:30:41] 10Continuous-Integration-Config: Move bundler Jenkins jobs to nodepool instances - https://phabricator.wikimedia.org/T114860#1708034 (10hashar) The Nodepool Jessie instances have ruby2.1 and bundler 1.7.4. We can create a new job template `bundle-{bundlecommand}-jessie` , will need a different builder macro bec... [09:16:51] (03Abandoned) 10Paladox: Add jshint to check: to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229178 (owner: 10Paladox) [10:09:03] !log deleted angry-caching-proxy.integration.eqiad.wmflabs | was an experiment no more use for it [10:09:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:13:08] PROBLEM - Host angry-caching-proxy is DOWN: CRITICAL - Host Unreachable (10.68.19.184) [10:26:57] 5Continuous-Integration-Scaling: Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1708261 (10hashar) I looked a bit at setting up a shared proxy using Squid3 and its ability to do SSL man in the middle (SSL-BUMP). That causes a few troubles, we would need to inject o... [10:42:53] 5Continuous-Integration-Scaling: Evaluate devpi for caching Pypi python packages - https://phabricator.wikimedia.org/T114871#1708265 (10hashar) 3NEW a:3hashar [10:53:28] 10Browser-Tests, 10Continuous-Integration-Config: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1708283 (10zeljkofilipin) @hashar: Why? I still think we should do it. [11:27:02] 5Continuous-Integration-Scaling: Evaluate devpi for caching Pypi python packages - https://phabricator.wikimedia.org/T114871#1708391 (10hashar) Setup ------- * `role::labs::lvm::srv` (to mount instance extended disk space at `/srv/`) * `contint::packages::python` (python, pip etc) * `pip install devpi-server`... [11:28:04] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Evaluate devpi for caching Pypi python packages - https://phabricator.wikimedia.org/T114871#1708393 (10hashar) [11:28:40] 5Continuous-Integration-Scaling: Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1708396 (10hashar) devpi for pip/python works out of the box and is straightforward {T114871} [11:28:59] zeljkof: the python dynamic cache is super easy to setup :-}) [11:29:08] https://phabricator.wikimedia.org/T114871#1708391 [11:43:39] 5Continuous-Integration-Scaling: Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1708440 (10hashar) Also found `Sonatype Nexus` which is a Java/Maven repository system. It seems to support proxying/caching for both npm and gem: | npm | https://books.sonatype.com/nexu... [11:59:38] 10Browser-Tests, 10Continuous-Integration-Config: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1708470 (10hashar) Was just wondering since that task has been opened back in October 2013. I guess the cucumber linting can be added to a fut... [12:00:02] 10Browser-Tests, 10Continuous-Integration-Config: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1708471 (10hashar) [12:00:03] 10Browser-Tests, 10Continuous-Integration-Config: Define an entry point for ruby material - https://phabricator.wikimedia.org/T104024#1708472 (10hashar) [12:29:38] (03PS2) 10Hashar: [SmashPig] add composer backed test suite [integration/config] - 10https://gerrit.wikimedia.org/r/243174 (https://phabricator.wikimedia.org/T104264) [12:29:45] (03CR) 10Hashar: [C: 032] [SmashPig] add composer backed test suite [integration/config] - 10https://gerrit.wikimedia.org/r/243174 (https://phabricator.wikimedia.org/T104264) (owner: 10Hashar) [12:31:02] (03Merged) 10jenkins-bot: [SmashPig] add composer backed test suite [integration/config] - 10https://gerrit.wikimedia.org/r/243174 (https://phabricator.wikimedia.org/T104264) (owner: 10Hashar) [12:32:51] 10Continuous-Integration-Config, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 5Patch-For-Review: Enable PHPUnit testing on the wikimedia/fundraising/SmashPig repo - https://phabricator.wikimedia.org/T104264#1708519 (10hashar) 5Open>3Resolved Seems all fine :-} [12:36:13] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: tox-jessie job is failing randomly on integration/config with could not install deps, because of throttling - https://phabricator.wikimedia.org/T114650#1708527 (10hashar) From yesterday meeting, that is Pypi causing some connection issue... [12:47:59] hashar: great :) [12:51:13] 10Continuous-Integration-Infrastructure, 6operations: Phase out operations-puppet-pep8 Jenkins job and tools/puppet_pep8.py - https://phabricator.wikimedia.org/T114887#1708567 (10hashar) 3NEW a:3hashar [12:54:27] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #619: 04FAILURE in 26 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/619/ [13:21:41] (03CR) 10Hashar: [C: 04-1] "Moaaar test. Please link to the related change in the source repo, that makes it easier to review / test :-}" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/239681 (owner: 10Paladox) [13:36:28] (03CR) 10Nikerabbit: Sniff to warn check usage of dirname(__FILE__) (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [13:37:11] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Clarify the salt version to use on beta cluster - https://phabricator.wikimedia.org/T114755#1708677 (10hashar) I have no idea why http://debian.saltstack.com/debian/ has been added. Seem it was a manual change related to testing the 2015.5... [13:41:10] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Clarify the salt version to use on beta cluster - https://phabricator.wikimedia.org/T114755#1708682 (10ArielGlenn) we run 2014.7.5 and we run it on precise through jessie both in production and in labs. 2015.5 runs nowhere and I have not e... [13:43:02] 10Gerrit-Migration, 10Wikidata: [Task] move git repositories that are dependencies of wikidata to gerrit - https://phabricator.wikimedia.org/T74907#1708687 (10JanZerebecki) [13:43:44] !log Cleaning up http://debian.saltstack.com/debian/ jessie-saltstack/main from some beta-cluster instances | https://phabricator.wikimedia.org/T114755 [13:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:52:44] !log Upgrading packages oon deployment-cache-{text04,mobile04,upload04} , downgrading salt-common and salt-minion in the process ( https://phabricator.wikimedia.org/T114755 ) [13:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:53:13] 10Browser-Tests, 10Continuous-Integration-Config: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1708706 (10zeljkofilipin) I have stumbled upon this task while cleaning up browser tests board. I was forgotten for a while, hence no activity. :( [13:54:15] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Clarify the salt version to use on beta cluster - https://phabricator.wikimedia.org/T114755#1708707 (10hashar) 5Open>3Resolved root@deployment-salt:~# salt '*' cmd.run 'grep -R salt /etc/apt/sources.list.d/' ``` lang=yaml deployment-cac... [13:57:22] !log fixing up apt on deployment-cache-parsoid04 as well as salt install https://phabricator.wikimedia.org/T114755 [13:57:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:58:24] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Clarify the salt version to use on beta cluster - https://phabricator.wikimedia.org/T114755#1708710 (10hashar) ``` # salt 'deployment-cache-*' pkg.version salt-common deployment-cache-mobile04.deployment-prep.eqiad.wmflabs: 2014.7.5+ds-1... [14:10:35] 10Continuous-Integration-Infrastructure, 7Upstream, 7Zuul: Change force merged cause a deadlock in Zuul gate-and-submit pipeline - https://phabricator.wikimedia.org/T93812#1708759 (10JanZerebecki) Besides the deadlock, Krinkle also mentioned on IRC that for future jobs using that repo from zuul it won't cont... [14:12:48] RECOVERY - Puppet staleness on deployment-cache-parsoid04 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:35:25] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #283: 04FAILURE in 7 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/283/ [14:38:32] hashar: yeah, that scap task was maybe a corner case for beta-cluster-infra vs -reproducible :) [14:42:51] (03CR) 10XZise: [IMPROV] Don't use shell argument (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/243140 (owner: 10XZise) [14:42:56] (03PS4) 10XZise: [IMPROV] Don't use shell argument [integration/jenkins] - 10https://gerrit.wikimedia.org/r/243140 [14:58:19] 10Continuous-Integration-Infrastructure: Jenkins needs the ability to sign tarballs - https://phabricator.wikimedia.org/T64391#1708920 (10greg) Yeah, and just for more context: to do this right we'd need support from Chris S and the Security team, which we won't get for a while anyways. It's just not a priority... [15:07:44] hashar_: https://github.com/hmlb/phpunit-vw we should enable this. [15:07:47] ;) [15:07:55] legoktm: ^ [15:08:06] hahaha [15:11:06] the projects build is passing, according to travis.... maybe [15:12:23] 5Continuous-Integration-Scaling, 7Nodepool, 5Patch-For-Review: Add monitoring and capacity planning for Nodepool - https://phabricator.wikimedia.org/T113806#1708976 (10hashar) [15:12:57] :-} [15:18:49] 6Release-Engineering-Team, 15User-greg: Investigate https://scrutinizer-ci.com/pricing WMF logo use - https://phabricator.wikimedia.org/T110396#1708996 (10greg) 5Open>3Resolved Response. Will be removed this week with their next deploy. They did suggest we could get their security scanning feature enabled... [15:44:04] 6Release-Engineering-Team, 6operations, 7Database: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1709107 (10jcrespo) 3NEW [15:45:03] !log Uncommited local change to /var/lib/git/operations/puppet/modules/scap/files/l10nupdate-1 on deployment-puppetmaster. Looks like something Krenair was working on. [15:45:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:45:37] Krenair: I'll make a real local commit out of that ^ [15:46:48] 6Release-Engineering-Team, 6operations, 7Database: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1709116 (10jcrespo) Due to the time that has passed since the values were lost, we are only going to reimport the missing values for now, to avoid annoying users a... [15:48:00] !log Made [LOCAL HACK] commit on deployment-puppetmaster for l10nupdate cache location change [15:48:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:49:30] !log Updated cherry-pick of https://gerrit.wikimedia.org/r/#/c/241984 on deployment-puppetmaster and forced puppet run on deployment-logstash2 [15:49:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:54:12] 6Release-Engineering-Team, 6operations, 7Database: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1709137 (10jcrespo) [16:03:15] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:15] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:15] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:06:34] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Figure out why PHPCS is unable to parse FormatMetadata.php - https://phabricator.wikimedia.org/T113871#1709195 (10TasneemLo) I'm setting the task status to resolved as the problem seems to have fixed itself \^_^/ [16:06:47] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown: Figure out why PHPCS is unable to parse FormatMetadata.php - https://phabricator.wikimedia.org/T113871#1709196 (10TasneemLo) 5Open>3Resolved [16:15:26] (03CR) 10TasneemLo: Sniff to warn check usage of dirname(__FILE__) (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [16:20:27] * marktraceur itches about beta scap taking a long time [16:20:34] Seems like every time I need it to run, it takes longer [16:21:41] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 7Beta-Cluster-reproducible: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1709238 (10greg) Code seems to be updating, at least? http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [16:22:03] marktraceur: heh, timing ^ [16:22:22] FFS [16:22:49] but code seems to be updating, at least, can you confirm that (since you seem to be waiting for specific code to be deployed) [16:23:11] I don't see my code on the site yet, no [16:23:32] Oh, and good, the messages are disappeared. [16:23:43] Worst. Demo. Ever. [16:24:07] I just hope it's not my fault [16:24:31] thcipriani: ^^ hate to ping you on this, but... scap failing in beta :( [16:26:57] greg-g: hmm...well that's been happening a while it looks like... [16:27:57] yeah, since last night [16:28:00] at least [16:28:10] The current run is taking a long time, though, that's not what the rest of them have been doing AFAICT [16:28:17] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/buildTimeTrend [16:34:30] Yippee, build fixed! [16:34:30] Project beta-scap-eqiad build #73226: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/73226/ [16:35:33] (03CR) 10Nikerabbit: Sniff to warn check usage of dirname(__FILE__) (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243096 (https://phabricator.wikimedia.org/T114267) (owner: 10TasneemLo) [16:36:35] (03PS1) 10Legoktm: Remove wmf/ branch exclusion for mediawiki-core-phpcs job [integration/config] - 10https://gerrit.wikimedia.org/r/244192 [16:36:50] Thanks thcipriani [16:36:51] (03PS2) 10Legoktm: Remove wmf/ branch exclusion for mediawiki-core-phpcs job [integration/config] - 10https://gerrit.wikimedia.org/r/244192 [16:37:39] marktraceur: wish I could take credit, fixed itself as I was looking into it. I think there was something corrupt in the tmp cdb file that it processes. [16:38:01] rebuilding on bastion must've fixed it. [16:39:38] huh [16:39:39] Fair enough [16:40:09] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 7Beta-Cluster-reproducible: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1709314 (10greg) ``` 16:37 <+thciprian> marktraceur: wish I could take credit, fixed itself as I was looking into it. I think... [16:40:24] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 7Beta-Cluster-reproducible: Scap failing with 'bool' object has no attribute 'encode' - https://phabricator.wikimedia.org/T114852#1709315 (10greg) 5Open>3Resolved [16:42:04] Still not seeing my code. Weird. [16:45:25] marktraceur: Add $wgForeignUploadTargets seems to be out on (at least one) of the targets. Is that the code you were waiting on? [16:57:32] marktraceur: ^ :) [17:02:57] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:21:38] 6Release-Engineering-Team, 6operations, 7Database, 5Patch-For-Review: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1709471 (10jcrespo) After inspecting the current state of the table, we believe that insert will not have any noticible effect, due to changes... [17:24:06] 6Release-Engineering-Team: Implement "WMF Log Errors count" KPI - https://phabricator.wikimedia.org/T108749#1709489 (10greg) a:3demon [17:24:23] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Zuul: Implement "Jenkins/Zuul queue wait" KPI - https://phabricator.wikimedia.org/T108750#1709491 (10greg) a:3hashar [17:28:44] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:44] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:44] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:36] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 44995 bytes in 0.675 second response time [17:36:31] 6Release-Engineering-Team, 6operations, 7Database, 5Patch-For-Review: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1709553 (10jcrespo) https://gerrit.wikimedia.org/r/#/c/244190/ was created for this task, and abandoned. Having a timestamp on user preferenc... [17:42:16] thcipriani: Sorry, yes, the code is out now [17:42:54] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:24] bd808, when should we be putting things in operations/puppet.git's hieradata/labs/deployment-prep instead of Hiera:Deployment-prep? [18:21:04] good question, I wouldn't mind knowing the thinking there either [18:21:33] Krenair: I think we had the wiki page before the labs hierarchy in operations/puppet [18:21:53] Krenair: I tend to prefer changes send to puppet.git then cherry picked on beta cluster puppetmaster [18:22:06] the wiki missing a blame command :-D [18:22:16] I put something on the wiki recently (host-specific Hiera namespace page) [18:22:24] But now I wonder if I should have put it in puppet.git [18:22:51] the nice thing with the wiki is you don't need to craft a git commit / figure out which /hieradata/ file to put your content in it [18:22:59] but there is some kind of overlaps between the two [18:27:33] and you don't need ops approval to make a labs change [18:33:07] Krenair: on beta lot of folks can cherry pick on the local puppetmaster [18:33:17] but yeah, doing a wiki edit is way simpler [18:33:18] until ops merges it? [18:51:35] 10Beta-Cluster-Infrastructure: Beta cluster bits should not cache static-master for three weeks - https://phabricator.wikimedia.org/T90983#1709839 (10matmarex) This is still the case and still just as frustrating. I forgot that it's been fucked up for months today and tried to test code on beta again. [19:17:50] Krenair: at one point the choice was about things that needed to be overridden per-host. The files in ops-puppet are always overridden by the on-wiki settings. Not that there is a per-host settings page on-wiki as well I suppose it is about convention and/or convenience [19:18:04] I would personally recommend that we put everything in ops-puppet [19:18:43] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, and 2 others: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1709909 (10chasemp) [19:19:20] the commit-in-gerrit + cherry-pick to deployment-puppetmaster combo works well for me on just about every change [19:19:32] bd808, ummmm [19:19:45] okay [19:19:49] so I made a deployment-bastion page on wikitech [19:19:58] the only time it sucks is when somebody nukes the cherry-picks and we have to recreate them from git history [19:21:37] this is really a question for releng folks since they actually own the environment [19:21:47] I'm just a left over helper from the before times :) [19:25:26] bd808: maybe we will use scap3 and handle the cherry picks like security patches for mw core [19:26:21] at least that largely simplified maintenance of the beta cluster [19:28:04] * bd808 imagines the scap(3) man page [19:28:12] scap as a system call :) [19:28:35] err I guess man(3) is C lib [19:48:16] 10Continuous-Integration-Infrastructure: handle composer require-dev together with mediawiki-vendor - https://phabricator.wikimedia.org/T112895#1710002 (10bd808) >>! In T112895#1650356, @Legoktm wrote: > I was thinking of: clone mediawiki/vendor, add use composer-merge-plugin to include mediawiki/core's composer... [20:07:28] 10Continuous-Integration-Infrastructure, 7Composer: Come up with non sucky solution for running "composer test" on repos that have vendor/ checked in - https://phabricator.wikimedia.org/T92605#1710110 (10bd808) >>! In T92605#1574097, @Tgr wrote: > Just put dev dependencies in gitignore? One could probably even... [20:10:24] 5Continuous-Integration-Scaling, 7Nodepool, 7Upstream: Nodepool delay instance deletions by one minute - https://phabricator.wikimedia.org/T113359#1710125 (10hashar) [20:11:04] 5Continuous-Integration-Scaling, 7Tracking: [tracking] Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1710131 (10hashar) [20:11:14] 5Continuous-Integration-Scaling, 7Tracking: [tracking] Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1638442 (10hashar) p:5Triage>3Normal [20:12:06] 5Continuous-Integration-Scaling, 7Nodepool: Nodepool should send metrics to statsd - https://phabricator.wikimedia.org/T111496#1710139 (10hashar) a:5hashar>3None Not currently working on it [20:14:48] 5Continuous-Integration-Scaling, 6Labs, 10Labs-Infrastructure: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#1710153 (10hashar) 5Open>3stalled p:5Low>3Lowest [20:14:50] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10releng-201415-Q3, 10releng-201415-Q4, 7Epic: [EPIC] Run CI jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#1710155 (10hashar) [20:15:44] 5Continuous-Integration-Scaling, 6Labs, 10Labs-Infrastructure: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#936078 (10hashar) If CI starts to cause troubles to other projects on labs, we will have to look at dedicated hardware for it. There is... [20:20:22] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Write a migration plan for CI infra to the disposable VMs infrastructure - https://phabricator.wikimedia.org/T86172#1710183 (10hashar) 5Open>3Invalid a:3hashar I filled that task after some random discussion we had were we envision... [20:53:19] twentyafterfour: re the arcanist 'staging-area' feature, is that pushed to from the committer's machine, or from diffusion/phabricator? (ie: who all needs permission to push to it? Can we restrict it to just Phab itself?) [20:53:23] s/ie/iow/ [20:53:59] greg-g: it is pushed by arcanist not phab. But it's a staging area so allowing pushes from anyone isn't harmful [20:54:12] (also, you allow pushes to a special ref space not to the main repo refs tree) [20:54:33] is that working now? [20:54:45] chasemp: yes as far as I know it is [20:54:51] it's been in for a while [20:55:10] and a lot of recent development upstream has been focused on that and harbormaster/drydock [20:55:18] I vaguely recall something missing / it was super half thought out at the time [20:55:34] interesting [20:55:56] I guess if we allow you to commit to master anyways :) [20:56:09] guys the battle for ssh git to phab continues, I'm pretty close now I think [20:56:16] I love you [20:56:47] the feeling is uncomfortably mutual [20:57:07] just the way I like it [21:01:52] lol [21:02:11] greg-g: 1:1? [21:02:28] 10Deployment-Systems, 15User-greg: Remove (?) l10nupdate from !log'ing - https://phabricator.wikimedia.org/T107480#1710374 (10greg) 5Open>3declined a:3greg Declining based on consensus. [21:04:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #789: 04FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/789/ [21:19:41] (03CR) 10Subramanya Sastry: "besides the inline comment, looks ok." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/243997 (https://phabricator.wikimedia.org/T92871) (owner: 10Hashar) [21:20:13] Yippee, build fixed! [21:20:14] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #27: 09FIXED in 4 min 13 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/27/ [21:34:58] 10Deployment-Systems, 6Release-Engineering-Team, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1710501 (10thcipriani) Following up on subpoints, point-by-point, expanding on T109535#1691326 > - rolling deploys / config changes... [21:49:13] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, and 2 others: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1710613 (10chasemp) Our ferm module doesn't seem to allow specification of the dst address at this moment. In this case we will have multiple SSH servi... [21:55:17] 3Scap3, 6Phabricator: Deploy phabricator with scap3 - https://phabricator.wikimedia.org/T114363#1710637 (10mmodell) [21:56:33] 10Deployment-Systems, 3Scap3: Implement nrpe config parser to easily hook into Puppet managed checks - https://phabricator.wikimedia.org/T114284#1710646 (10mmodell) 5Open>3Resolved [22:11:53] twentyafterfour: can you add #deployment-systems to https://phabricator.wikimedia.org/diffusion/GSV/ ? kthxbai [22:12:09] * greg-g 'd just do it but I'm not a repo-admin [22:12:49] greg-g: done [22:17:59] PROBLEM - Puppet failure on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:22:37] 6Release-Engineering-Team, 6operations, 7Database, 5Patch-For-Review: Recover missing values from user_properties tables - https://phabricator.wikimedia.org/T114899#1710873 (10demon) p:5Triage>3Low [22:30:54] 10Deployment-Systems, 5Patch-For-Review: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server - https://phabricator.wikimedia.org/T104826#1710938 (10bd808) a:5bd808>3None [22:39:28] PROBLEM - Puppet failure on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:56:09] Yippee, build fixed! [22:56:09] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #823: 09FIXED in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/823/ [23:25:22] 10Deployment-Systems, 3Scap3: Implement nrpe config parser to easily hook into Puppet managed checks - https://phabricator.wikimedia.org/T114284#1711082 (10dduvall)