[00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:54:14] 10Phabricator: Email sometimes not being sent when a task is created - https://phabricator.wikimedia.org/T182549#4119064 (10Anomie) Unfortunately "other activity" includes a lot of crap I really don't care to get mail about. [01:46:41] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119091 (10awight) I ran a test on deployment-ores01.deployment-prep.eqiad... [01:47:59] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119092 (10awight) [02:20:32] Project beta-scap-eqiad build #203190: 04FAILURE in 4 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/203190/ [02:25:36] legoktm, hi, i think there's a bug in the extension registration system, when you try and overide the Math extension defaults with $wgMathValidModes = [ 'png' ]; it does not work [02:25:40] (this is in 1.30) [02:26:01] editing the extension directly and changing it in extension.json works. [02:26:20] but that should be considered a work around as you should be able to change the defaults through the config [02:29:06] Yippee, build fixed! [02:29:07] Project beta-scap-eqiad build #203191: 09FIXED in 5 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/203191/ [02:31:30] legoktm it seems to be merging in png twice [02:31:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [02:31:36] see https://phabricator.wikimedia.org/P6971 [02:31:44] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4119112 (10mmodell) @atgo: {F16889710} [02:32:11] paladox: there's a bug filed somewhere for it [02:32:18] ah [02:47:07] 10Phabricator: Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865#4119121 (10mmodell) [02:48:37] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119135 (10mmodell) >>! In T181071#4118685, @awight wrote: > I can't tell... [02:52:41] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119136 (10mmodell) >>! In T181071#4119091, @awight wrote: > There's no .... [02:56:22] legoktm: do you have the link please? :) [02:56:25] I’ve looked [02:56:42] But found nothing in the titilematching the problem [02:56:47] We are having [02:58:45] paladox: https://phabricator.wikimedia.org/T150011 [02:59:13] Thanks [02:59:21] legoktm: is this similar to https://phabricator.wikimedia.org/T186785 ? [03:03:05] well that was one idea on how to implement it [03:03:07] which won't work [03:04:12] Oh ok [03:38:14] Project mediawiki-core-code-coverage-php7 build #197: 04STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/197/ [04:25:17] Project mediawiki-core-code-coverage build #3436: 04STILL FAILING in 1 hr 25 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3436/ [04:38:12] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119223 (10awight) The weird part about this is just that we've been deplo... [05:20:04] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#4119262 (10Marostegui) [06:31:06] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#4119331 (10Marostegui) [07:06:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:25:17] RECOVERY - Free space - all mounts on integration-slave-docker-1006 is OK: OK: All targets OK [07:31:41] Project mwext-phpunit-coverage-publish build #3174: 04FAILURE in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3174/ [07:34:18] Yippee, build fixed! [07:34:18] Project mwext-phpunit-coverage-publish build #3175: 09FIXED in 2 min 2 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3175/ [08:05:01] o/ [08:05:44] zeljkof: o/ [08:06:27] Any idea about https://phabricator.wikimedia.org/T191537 then? :D [08:08:06] addshore: sorry, didn't have the time to take a look yet [08:08:16] Will do today [08:08:19] sweeet [08:08:28] if you need any help from our side feel free to ping me :) [08:11:01] * addshore adds a irc cloud avatar too [08:52:44] 10Beta-Cluster-Infrastructure: Create mediawiki::maintenance server (aka terbium) in deployment-prep - https://phabricator.wikimedia.org/T187826#4119519 (10EddieGP) [08:52:46] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#4119522 (10EddieGP) [08:53:11] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Remove video scaler instances from deployment-prep - https://phabricator.wikimedia.org/T187063#3963166 (10EddieGP) According to openstack browser, both instances are gone. Is this resolved? [08:53:54] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Remove video scaler instances from deployment-prep - https://phabricator.wikimedia.org/T187063#4119525 (10MoritzMuehlenhoff) 05Open>03Resolved a:03MoritzMuehlenhoff Ack, I removed those last week, closing the task. [08:56:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Recommendation-API, 10Patch-For-Review, 10Scoring-platform-team (Current): What to do with deployment-sca03? - https://phabricator.wikimedia.org/T184501#4119535 (10EddieGP) 05Open>03Resolved a:03EddieGP That instance... [09:12:00] 10Phabricator: Email sometimes not being sent when a task is created - https://phabricator.wikimedia.org/T182549#4119572 (10EddieGP) That's why I didn't mark the task resolved. But I guess we now have an idea of where this comes from and what to report to upstream: "Please move task creation notifications out of... [09:18:40] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4119585 (10ze... [09:43:19] 10Phabricator: Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865#4119640 (10Aklapper) Kind of a dup of T76094 ? [09:44:43] 10Phabricator: Email sometimes not being sent when a task is created and "other task activity" is not set in user preferences - https://phabricator.wikimedia.org/T182549#4119661 (10Aklapper) [09:46:17] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<55.56%) [10:49:28] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4119832 (10Aklapper) What is a "goal" and what does "under" mean exactly? :) Also see https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milest... [10:50:16] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4119834 (10Aklapper) (For the records, a [[ https://phabricator.wikimedia.org/project/members/1776/ | good bunch of CC'ed people should be able to create this ]] once it's clearer what's... [11:04:18] 10Gerrit, 10Release-Engineering-Team (Someday): Update gerrit to 2.15.1 - https://phabricator.wikimedia.org/T177201#4119849 (10Paladox) [12:38:15] !log gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package | T191771 [12:38:15] [12:38:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:38:20] T191771: [REL1_30] Some parserTests fail on debian stretch using Tidy, because of a new version of libtidy - https://phabricator.wikimedia.org/T191771 [12:38:46] (03PS1) 10Hashar: Add debian-glue to operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425256 (https://phabricator.wikimedia.org/T191771) [12:38:58] (03CR) 10Hashar: [C: 032] Add debian-glue to operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425256 (https://phabricator.wikimedia.org/T191771) (owner: 10Hashar) [12:40:10] (03Merged) 10jenkins-bot: Add debian-glue to operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425256 (https://phabricator.wikimedia.org/T191771) (owner: 10Hashar) [13:20:49] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:46] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36492 bytes in 5.756 second response time [13:40:38] (03CR) 10Krinkle: [C: 032] Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [13:41:23] !log upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854) [13:41:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:41:26] T184854: hhvm memcached and php7 memcached extensions do not play well together - https://phabricator.wikimedia.org/T184854 [13:41:33] (03Merged) 10jenkins-bot: Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [13:42:21] (03CR) 10jenkins-bot: Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [13:48:46] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday), 10Test-Coverage: Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#4120208 (10Krinkle) @Legoktm Now that we have the mediawiki-core-code-coverage-php7 job and it's passing as well as the mediawi... [13:48:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday), 10Test-Coverage: Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#4120209 (10Krinkle) p:05Low>03High [13:50:30] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support: Make Wikimedia CI run PHP in either PHP 7.0+ or HHVM - https://phabricator.wikimedia.org/T190547#4076530 (10Krinkle) [13:50:34] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#4120211 (10Krinkle) 05Open>03stalled This should be the last thing to happen in the c... [13:59:38] Project mwext-phpunit-coverage-publish build #3191: 04FAILURE in 3.9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3191/ [14:01:26] Yippee, build fixed! [14:01:26] Project mwext-phpunit-coverage-publish build #3192: 09FIXED in 1 min 38 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3192/ [14:01:56] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4120256 (10ssastry) Well, I was looking at https://www.mediawiki.org/wiki/Phabricator/Project_management :-) .. and I assumed it is like a subproject and hence can be nested in a project.... [14:04:39] Project mwext-phpunit-coverage-publish build #3193: 04FAILURE in 3.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3193/ [14:05:48] Yippee, build fixed! [14:05:48] Project mwext-phpunit-coverage-publish build #3194: 09FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3194/ [14:12:07] no_justification that was quick they have released 2.15.1 already heh [14:12:19] last time it took them a while to release 2.14.1 heh [14:16:01] Heh [14:17:01] no_justification already has alot of fixes [14:17:02] https://www.gerritcodereview.com/releases/2.15.md#2.15.1 [14:20:00] Allow graceful rolling restarts [14:20:00] Set a graceful stop timeout for allowing Jetty to wait for incoming requests to be completed before shutting down its sockets. [14:20:08] Holy crap I've wanted that forever. [14:20:24] Basically "stop taking new connections, but don't kill folks who are in flight already" [14:20:24] lol [14:20:34] luca added that [14:20:40] <3 [14:20:40] i presume for gerrithub [14:20:42] I owe him a beer. [14:20:52] lol [14:21:46] https://github.com/GerritCodeReview/gerrit/commit/4e2bb4562c316de80b796575a7b16a8b56da48b4 [14:34:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (MediaWiki-MultiDC): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#4120368 (10Joe) [14:36:18] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [14:36:23] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4120369 (10Aklapper) Note the differences between subprojects vs milestones when it comes to cards being shown on parent workboards and project members being moved when creating. :) Whate... [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:01:24] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday), 10Test-Coverage: Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#4120471 (10Legoktm) Yeah, once T191863 is fixed, we can drop the php5 one, and rename the php7 one to the standard name. [15:06:16] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [15:09:21] ok, I must have overlooked something stupid [15:09:22] but [15:09:39] https://gerrit.wikimedia.org/r/#/c/425272/3/hieradata/labs/deployment-prep/common.yaml I put deployment-snapshot01 back in for scap and yet, no scaps happening [15:12:24] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:39] the change didn't make it over to deployment-puppetmaster02 [15:14:23] merged an hour ago [15:16:30] maybe there's merge conflicts on the puppetmaster? [15:16:39] git fetch origin && git rebase origin [15:16:45] in /var/lib/git/operations/puppet [15:16:50] apergos ^^ [15:17:01] * eddiegp looks [15:17:15] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [15:17:55] maybe, I haven't checked that yet, but I thought I would see those in the logs somewhere [15:18:37] Yeah, rebase conflict [15:18:43] oh great, thanks [15:18:53] there's a bunch of other cruft in the log that makes it hard to sort out [15:18:58] Apr 10 15:18:13 deployment-puppetmaster02 puppet-master[25019]: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, unable to init lv-a for swift at /etc/puppet/modules/swift/manifests/init_device.pp:3:9 at /etc/puppet/modules/role/manifests/swift/storage.pp:23 on node deployment-ms-be03.deployment-prep.eqiad.wmflabs and like that [15:19:10] Apr 10 15:18:10 deployment-puppetmaster02 puppet-master[25019]: Unknown variable: '::projectgroup'. at /etc/puppet/manifests/realm.pp:46:8 [15:19:10] [15:19:12] blah blah [15:19:19] oh [15:19:42] tail /var/log/git-sync-upstream.log on puppetmaster [15:19:44] i think there's a task for deployment-ms-be03.deployment-prep.eqiad.wmflabs [15:19:47] some where [15:19:48] will let you know if rebase failed [15:20:17] ah that's the file, i forgot where it was [15:20:19] aaand it looks like it is failing [15:20:41] meeeehhhh [15:21:39] will there be any deploy freeze around the hackathon? [15:25:24] apergos: looks like https://gerrit.wikimedia.org/r/#/c/425077/ may be both cherry picked and merged, maybe just need to remove it [15:27:07] sure, the chery icked one is ealier than the merged one so do it [15:27:12] *cherry picked [15:27:21] thcipriani: [15:27:24] * thcipriani does [15:27:32] *earlier [15:27:41] that must be a record for most typos in a line [15:29:42] ah, and https://gerrit.wikimedia.org/r/#/c/425208/ removed both cherry-picks, looks up-to-date now [15:29:58] well. rebased successfully anyway [15:31:13] looks better [15:34:19] now I need to wait for the sync job.... [15:34:28] er scap job [15:34:53] thcipriani: Could you have a look at https://phabricator.wikimedia.org/T190755 ? I saw it's gone unnoticed while all the other access (restore) request were handled. [15:35:46] It does 'scap sync', so ... ;) [15:37:08] Project mediawiki-core-code-coverage-php7 build #198: 04STILL FAILING in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/198/ [15:37:18] addshore: I don't see why we'd need to, really? [15:38:00] Long as not everyone from Releng and Ops are gonna be there (which I know to be the case) [15:38:12] legoktm: https://hackernoon.com/generating-code-coverage-with-phpunite-and-phpdbg-4d20347ffb45 [15:38:26] legoktm: Interesting. phpdbg significantly outperforms php+xdebug [15:38:31] Worth investigating? [15:48:41] PROBLEM - Puppet errors on integration-slave-docker-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:49:18] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:56:33] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/203270/console very very very slow [15:56:57] 15:36:08 15:36:08 Updating LocalisationCache for master using 6 thread(s) [15:57:19] yeah [15:58:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for access to the beta cluster - https://phabricator.wikimedia.org/T190755#4082855 (10thcipriani) I added users `chelsyx` and `bearloga` to the deployment-prep project. ssh to deployment-prep/beta should authenticate you now. The connection cl... [15:59:06] I...bet that slowness has to do with using hhvm for mwscript again [16:07:31] the previous jobs were all faster [16:12:08] yeah, they may have no needed actual l10nupdates though, this last deploy definitely touched /extensions/Translate/i18n/api/en.json also a few builds back there was a 36 minute one. [16:25:06] Project mediawiki-core-code-coverage build #3437: 04STILL FAILING in 1 hr 25 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3437/ [16:27:52] ah that's probably it [16:27:53] groan [16:28:08] anyways the scap finally reached my instance. [16:34:16] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#4120740 (10zeljkofilipin) 05Open>03stalled [16:35:53] 10Project-Admins, 10Africa-Wikimedia-Developers: Project work board request for WikiFundi - https://phabricator.wikimedia.org/T186754#4120749 (10Aklapper) 05Open>03stalled p:05Normal>03Lowest Alright, I am going to set the status to `stalled` here for the time being. Once a decision has been made, plea... [16:39:17] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:43] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120819 (10awight) >>! In T181071#4119135, @mmodell wrote: >>>! In T181071... [17:13:40] (03Abandoned) 10Zfilipin: WIP Selenium: record video [integration/config] - 10https://gerrit.wikimedia.org/r/422949 (https://phabricator.wikimedia.org/T179188) (owner: 10Zfilipin) [17:15:21] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120858 (10awight) @mmodell: We're running the fetch check with "bash -x"... [17:34:51] no_justification https://gerrit-review.googlesource.com/c/gerrit/+/166850 :) [17:35:11] and https://gerrit-review.googlesource.com/c/gerrit/+/170454 [17:37:39] oh wow they merged alot of my backports today :) [17:39:28] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120939 (10awight) There's a wealth of surprising results on tin, /srv/dep... [17:45:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for access to the beta cluster - https://phabricator.wikimedia.org/T190755#4120969 (10chelsyx) Thanks @thcipriani ! [17:51:31] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927#4120986 (10EddieGP) [17:55:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: fr.wikipedia.beta.wmflabs.org uses an invalid security certificate - https://phabricator.wikimedia.org/T188288#4121015 (10EddieGP) 05Open>03Resolved The puppet errors are fixed and `fr.wikipedia.beta.wm... [18:15:30] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4121085 (10Dzahn) per the last ops meeting and joe's comments: - reinstall it one more time. back to stretch instead of jessie [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:05:57] PROBLEM - SSH on integration-slave-docker-1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:50] RECOVERY - SSH on integration-slave-docker-1003 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [19:11:53] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121339 (10mmodell) >>! In T181071#4120939, @awight wrote: > It would be g... [19:12:28] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121341 (10mmodell) [19:12:36] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:27] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [19:21:53] (03PS1) 10Umherirrender: Make mwext-PoolCounter-rake-docker non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/425337 [19:23:08] (03CR) 10jerkins-bot: [V: 04-1] Make mwext-PoolCounter-rake-docker non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/425337 (owner: 10Umherirrender) [19:28:49] (03CR) 10Umherirrender: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/425337 (owner: 10Umherirrender) [19:30:00] (03CR) 10jerkins-bot: [V: 04-1] Make mwext-PoolCounter-rake-docker non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/425337 (owner: 10Umherirrender) [19:42:08] no_justification wondering could you review https://gerrit.wikimedia.org/r/#/c/424710/ and https://gerrit.wikimedia.org/r/#/c/424708/ please :). (the last one could be merged when ever dosen't require the plugin to be installed first) [19:42:48] I have no time for that right now. [19:43:06] ok [19:45:22] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4121389 (10ssastry) 05Open>03Resolved a:03ssastry James F created https://phabricator.wikimedia.org/project/profile/3331/ for us. [19:52:38] Krinkle: iirc the response from Sebastian Bergmann was that phpdbg was less accurate than xdebug [19:54:54] 10Phabricator: Email sometimes not being sent when a task is created and "other task activity" is not set in user preferences - https://phabricator.wikimedia.org/T182549#4121423 (10mmodell) Started a discussion upstream: https://discourse.phabricator-community.org/t/is-task-creation-notified-under-other-task-act... [19:56:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T183968#4121437 (10thcipriani) [20:33:45] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4121553 (10atgo) @mmodell this is great! One thought... it might make more sense to have tasks that have a longer deadline be something neutral/not eye catching, and then escalate to orange and then red as they get closer. The i... [20:52:46] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#4121599 (10MarcoAurelio) Is this something that requires an ammount of non-trivial work? Otherwise we can list in a page somewhere in Wikitech which scripts should be r... [20:56:18] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4121602 (10mmodell) >>! In T93499#4121553, @atgo wrote: > @mmodell this is great! One thought... it might make more sense to have tasks that have a longer deadline be something neutral/not eye catching, and then escalate to oran... [20:59:52] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#2001990 (10Dzahn) I suggest to create a fresh instance (that is not named after a hostname in prod but has a generic name) and apply role(mediawiki_maintenance) to it.... [21:02:36] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#4121613 (10MarcoAurelio) @Dzahn Thanks for your explanation. I agree with the naming, etc. As for "see which erros you actually get" I'm afraid I'd not be able to do so... [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:24:55] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#4121647 (10MarcoAurelio) [21:26:19] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4121650 (10thcipriani) [21:26:36] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#2001990 (10MarcoAurelio) Also, what about `deployment-maintenance` with `role::mediawiki_maintenance`? [21:34:26] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865#4121724 (10mmodell) a:03mmodell [21:38:41] (03PS1) 10Thiemo Kreuz (WMDE): Remove empty lines from comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425423 [21:44:29] (03PS1) 10Thiemo Kreuz (WMDE): Use strrpos() to look for newlines [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425425 [21:49:23] (03PS1) 10Thiemo Kreuz (WMDE): Replace strpos() with faster substr() comparisons [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425426 [21:54:25] (03PS1) 10Thiemo Kreuz (WMDE): Shorten out earlier in the DbrQueryUsage sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425427 [21:59:04] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4121780 (10atgo) Thanks @mmodell. Bummer about the notifications, thanks for letting me know. If I might make a suggestion, this might be more intuitive, since I'd think a missed deadline would be something folks would want to... [21:59:40] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4121781 (10atgo) Also, what is the behavior if no deadline is set? IMO it should display nothing there. [22:01:50] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#2001990 (10Krinkle) 👍 [22:02:37] (03PS1) 10Thiemo Kreuz (WMDE): Minor performance optimizations to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425429 [22:04:40] (03PS1) 10Thiemo Kreuz (WMDE): Shorten out earlier in the FunctionComment sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425431 [22:07:38] (03PS1) 10Thiemo Kreuz (WMDE): Scan for return tags from the end of the function scope [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425432 [22:16:13] (03PS1) 10Thiemo Kreuz (WMDE): Faster scan for namespaces in the PrefixedGlobalFunctions sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425434 [22:43:34] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4121912 (10thcipriani) Profiling info for rebuilding a single file via the command: `mwscript rebuildLocalisationCache.php --wiki=enwiki --outdir=/tmp... [22:44:59] (03CR) 10Krinkle: [C: 031] Minor performance optimizations to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425429 (owner: 10Thiemo Kreuz (WMDE)) [22:50:38] (03CR) 10Krinkle: [C: 032] Shorten out earlier in the FunctionComment sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425431 (owner: 10Thiemo Kreuz (WMDE)) [22:50:52] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4121932 (10mmodell) What could it be wait4ing for? [23:03:47] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121947 (10awight) Back to the drawing board. Including the packaged Pyth... [23:05:02] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121949 (10mmodell) @awight: why not build the virtualenv on a developer m... [23:07:45] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121952 (10awight) @mmodell: That would be wonderful, but virtualenvs are... [23:09:35] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121953 (10mmodell) So why even use virtualenv if they are based on site p... [23:17:47] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121994 (10awight) That's right, we do use the --system-site-packages flag... [23:55:06] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4122066 (10thcipriani) >>! In T191921#4121932, @mmodell wrote: > What could it be wait4ing for? Probably a red herring in this instance. There are 7 `... [23:59:51] (03PS1) 10Legoktm: Add phan for multiple extensions [integration/config] - 10https://gerrit.wikimedia.org/r/425446