[00:20:39] 10Continuous-Integration-Config: Puppet failing with labs ci class applied - https://phabricator.wikimedia.org/T182642#3829777 (10Paladox) [00:26:34] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3829796 (10Catrope) [00:27:41] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3829817 (10thcipriani) [00:28:39] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3829796 (10thcipriani) This is fixed in scap's master branch. Don't know when a new version will go out, but this change will go out with it. [01:09:52] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3829873 (10Huji) @hashar: Currently MediaWiki-Codesniffer does not have a rule for this indentation, does it? If so, would you expect me to first submit a patch to that project to introduce the... [01:33:40] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826816 (10Legoktm) I think it's fine to do both in parallel. I think hashar was just pointing out that if there's no codesniffer rule to enforce it, it will easily regress. [02:36:27] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3826816 (10Krinkle) >>! In T182546#3829896, @Legoktm wrote: > I think it's fine to do both in parallel. I think hashar was just pointing out that if there's no codesniffer rule to enforce it, i... [02:37:20] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3829948 (10Legoktm) Data was provided in the linked discussion: https://www.mediawiki.org/w/index.php?title=Topic:U39uajst25fusqgk [02:39:45] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3829949 (10Krinkle) I don't know if we're doing a poll of any kind, but based on Ed's assessment I'd now favour towards indenting (consistency with JS, and because it's a curly brace block of w... [02:53:41] Project beta-scap-eqiad build #185942: 04FAILURE in 0.35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185942/ [03:13:49] Yippee, build fixed! [03:13:49] Project beta-scap-eqiad build #185943: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/185943/ [04:50:18] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:50:44] Anyone know why: https://www.mediawiki.org/wiki/MediaWiki_1.31/wmf.5 onward release pages are empty? [04:50:50] greg-g: ^ [04:52:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [05:22:54] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:30:30] kart_: dunno. no_justification you've been running make-release-notes, right? :) [05:52:15] Eh I slacked. The script was having issues and I hadn't run them down. Plus someone (I forget whom) was always doing them when I forgoy [06:01:11] no_justification: OK. It will be great to bring it back. [06:01:38] AFAIK, Elena and sometime me checks it against features being deployed in production for QA. [06:13:53] Tbh, this should be easy to automate, idk why I never have [06:21:56] (03PS1) 10SaikrishnaS: Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/397738 [06:26:38] hmm ^ doesn't look right [06:28:35] (03CR) 10MarcoAurelio: [C: 04-1] "This doesn't look right." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/397738 (owner: 10SaikrishnaS) [06:31:06] wmf.11 uploaded [06:31:52] (03CR) 10Chad: [C: 04-2] "What are you trying to accomplish here?" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/397738 (owner: 10SaikrishnaS) [06:39:59] hi no_justification -- that user only have -2ed patches :/ [06:51:55] Project selenium-Wikibase ยป chrome,beta,Linux,BrowserTests build #572: 04FAILURE in 2 hr 11 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/572/ [06:55:17] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:04:46] 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10MW-1.30-release, 10Release: Release MediaWiki 1.30 - https://phabricator.wikimedia.org/T169127#3830079 (10greg) a:03demon [07:32:07] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 10Language-2017-Oct-Dec: [betalabs] Special:Translate does not allow to translate article content - multiple "Uncaught SyntaxError: Unexpected token < in JSON at position 0" - https://phabricator.wikimedia.org/T180841#3830105 (10KartikMistry) [07:32:43] hashar: ^^ Need to repopulate tables for translate or I'm missing something. Please check when you're free :) [07:37:05] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [07:46:30] kart_: i am around :) [07:47:47] kart_: the databases are updated via update.php , so usually the schema is kept up to date [08:00:24] Does one need any advanced permissions to run the puppet compiler? I've tried to follow https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler for a patchset of mine, but jenkins tells me "Access denied". [08:00:45] I mean I can probably do without, but it made me wonder what would be needed. [08:12:57] eddiegp: I dont think you can build a job indeed [08:14:35] eddiegp: yup one has to be in the "nda" group [08:15:26] eddiegp: but I can run it for you I guess [08:15:44] hashar: Hmm, okay, thanks. [08:16:38] That's be nice. I wanted to try on gerrit 394902 :) [08:17:24] *That'd [08:26:18] (03PS1) 10Chad: make-release-notes: Also allow passing the user/password from the CLI [tools/release] - 10https://gerrit.wikimedia.org/r/397744 [08:26:36] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3830208 (10hashar) [08:28:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Automatically publish python modules to pypi on tag publishing - https://phabricator.wikimedia.org/T182657#3830216 (10hashar) [08:37:40] eddiegp: against which hosts ? [08:38:07] oh that is varnish bah [08:38:41] hashar: Any varnish host should be fine, I guess it should apply to all of them in the same way. [08:38:53] yeah there is no need to puppet compile that imho [08:39:41] eddiegp: they have some test suites as well eg modules/varnish/files/tests/text/08-mobile-hostnames-rewrite.vtc [08:39:44] but I have no idea how to run them [08:40:10] guess Brandon / Ema will do the review and add a test if needed [08:41:02] hashar: Yeah, as I said, I can probably do without it on this one. :) Just made me curious whether/how one could use the compiler. [08:49:44] eddiegp: basically, the job/instance has facts from production hosts [08:50:02] it would then compile a catalog for the "production" branch and the last patchset of the proposed change [08:50:09] foreach of the hosts mentionned [08:50:25] then report the failures or the difference between "production" branch and the change [09:00:15] hashar: Thanks :) [09:33:31] (03CR) 10Dalba: "- https://integration.wikimedia.org/ci/job/pywikibot-core-tox-doc-docker/130/console" [integration/config] - 10https://gerrit.wikimedia.org/r/397544 (owner: 10Dalba) [10:01:40] 10Continuous-Integration-Config, 10Composer, 10Patch-For-Review: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3830378 (10hashar) There should not be any proxy intercepting traffic between the labs instance and the internet. As I know it... [10:10:42] (03CR) 10Addshore: "How do we test build these now? :)" [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) (owner: 10Hashar) [10:32:46] 10Continuous-Integration-Config, 10Composer, 10Patch-For-Review: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3830406 (10hashar) Files repo-old/https---packagist.org/packages.json and repo/https---packagist.org/packages.json differ File... [10:41:54] (03CR) 10Hashar: "https://www.mediawiki.org/wiki/Continuous_integration/Docker#Images_using_docker-pkg" [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) (owner: 10Hashar) [10:54:00] 10Release-Engineering-Team, 10Deployments, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3830443 (10Ladsgroup) Sure [11:06:28] (03CR) 10Hashar: [C: 032] "Indeed yes. I have refreshed the job, thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/397544 (owner: 10Dalba) [11:07:52] (03Merged) 10jenkins-bot: Increase pywikibot-core-tox-doc-docker timeout to 10 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/397544 (owner: 10Dalba) [11:15:41] hashar hi, could you take a look at https://gerrit.wikimedia.org/r/#/c/397601/ (https://phabricator.wikimedia.org/T182642) please? [11:42:35] (03PS1) 10Hashar: Ensure xdebug.so is loaded for phpunit coverage [integration/config] - 10https://gerrit.wikimedia.org/r/397778 (https://phabricator.wikimedia.org/T175028) [11:43:00] paladox: oh my .... :D [11:43:16] yep :) [11:46:32] paladox: I have to dig a bit more [11:46:36] ok [11:52:57] paladox: so the permanent jessie slaves still have ci::slave::browsertests [11:53:01] but afaik they no more need it [11:53:06] oh [11:53:12] im using the main role [11:53:37] role::ci::slave::labs [11:54:16] hmm selenium-Wikibase-chrome [11:55:49] hashar im guessing it's when you switched from a include to class [11:56:00] yup [11:56:17] https://gerrit.wikimedia.org/r/#/c/392956/3/modules/contint/manifests/browsertests.pp [11:56:22] https://gerrit.wikimedia.org/r/#/c/392956/3/modules/profile/manifests/ci/browsertests.pp [11:56:31] i wonder would using a require fix the problem? [12:05:35] paladox: slightly amended :) [12:05:51] eventually most of that will be scrapped / refactored [12:06:43] (03CR) 10Hashar: [C: 032] Ensure xdebug.so is loaded for phpunit coverage [integration/config] - 10https://gerrit.wikimedia.org/r/397778 (https://phabricator.wikimedia.org/T175028) (owner: 10Hashar) [12:07:53] (03Merged) 10jenkins-bot: Ensure xdebug.so is loaded for phpunit coverage [integration/config] - 10https://gerrit.wikimedia.org/r/397778 (https://phabricator.wikimedia.org/T175028) (owner: 10Hashar) [12:08:37] paladox: so that is a left over [12:08:42] I gotta move out for an hour or so [12:08:59] but I guess puppet will be fixed now. Thanks ! [12:21:20] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [12:31:29] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [12:31:39] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [12:43:01] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [12:47:40] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:52:25] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3830754 (10Huji) Okay, so plan of action is: 1) Add this feature to CodeSniffer 2) Modify the current submitted patch to also include a bump in the codesniffer version Correct? [13:05:50] (03CR) 10Florianschmidtwelzow: [C: 032] make-release-notes: Also allow passing the user/password from the CLI [tools/release] - 10https://gerrit.wikimedia.org/r/397744 (owner: 10Chad) [13:06:23] (03Merged) 10jenkins-bot: make-release-notes: Also allow passing the user/password from the CLI [tools/release] - 10https://gerrit.wikimedia.org/r/397744 (owner: 10Chad) [13:08:57] hashar thanks :) [13:10:39] hashar works [13:10:39] :) [13:22:39] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [13:34:01] paladox: yup :) [13:55:17] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] [14:05:17] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] [14:09:09] (03PS3) 10Hashar: docker: provision operations-puppet as 'nobody' [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) [14:09:27] (03CR) 10Hashar: [C: 032] docker: provision operations-puppet as 'nobody' [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) (owner: 10Hashar) [14:10:38] (03Merged) 10jenkins-bot: docker: provision operations-puppet as 'nobody' [integration/config] - 10https://gerrit.wikimedia.org/r/397577 (https://phabricator.wikimedia.org/T178620) (owner: 10Hashar) [14:22:28] (03PS1) 10Hashar: docker: fix typo in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/397818 [14:31:07] 10Release-Engineering-Team (Kanban), 10MediaWiki-Debian, 10User-zeljkofilipin: Upgrade RuboCop and Rubyzip in mediawiki/debian - https://phabricator.wikimedia.org/T182401#3831051 (10zeljkofilipin) a:03zeljkofilipin [14:40:04] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3831077 (10hashar) 05Open>03Resolved [14:43:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3831107 (10hashar) Tested on https://gerrit.wikimedia.org/r/#/c/397817/ That also fix `/cache/` b... [14:49:18] (03PS1) 10Zfilipin: Ignore Gemfile.lock [tools/release] - 10https://gerrit.wikimedia.org/r/397832 (https://phabricator.wikimedia.org/T182401) [14:51:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-Debian, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade RuboCop and Rubyzip in mediawiki/debian - https://phabricator.wikimedia.org/T182401#3831147 (10zeljkofilipin) @Legoktm Thanks a lot for the quick reply! I am fine with resolving/declining the task as... [15:03:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Patch-For-Review: Disable xdebug for phpunit/composer unless needed - https://phabricator.wikimedia.org/T175028#3831208 (10hashar) From T178701, `composer-package-php70-docker` has XDebug enabled. The jobs running HHVM or Zen... [15:05:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Enable Xdebug on Jenkins to calculate code coverage. - https://phabricator.wikimedia.org/T178701#3700120 (10hashar) XDebug is disabled by default since that slow down PHPUnit / composer etc. There are a few cases w... [15:06:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691#3831221 (10zeljkofilipin) [15:06:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691#3831231 (10zeljkofilipin) p:05Triage>03Low [15:07:22] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3831233 (10zeljkofilipin) [15:07:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691#3831221 (10zeljkofilipin) [15:15:02] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:19:49] (03PS1) 10Addshore: parameter_functions load EventLogging with TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/397840 [15:20:00] (03CR) 10Addshore: [C: 032] parameter_functions load EventLogging with TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/397840 (owner: 10Addshore) [15:20:25] hashar: ^^ just double checking that one deploy itself? [15:21:07] (03Merged) 10jenkins-bot: parameter_functions load EventLogging with TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/397840 (owner: 10Addshore) [15:27:36] !log unblocked beta scaps and files syncs on jenkins [15:27:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:29:49] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#3831274 (10zeljkofilipin) [15:30:00] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3831285 (10zeljkofilipin) [15:30:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#3831284 (10zeljkofilipin) [15:30:16] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#3831274 (10zeljkofilipin) p:05Triage>03Low [15:31:30] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#3831274 (10zeljkofilipin) [15:33:18] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Investigate replacing nodemw with mwbot - https://phabricator.wikimedia.org/T181284#3831294 (10zeljkofilipin) p:05Normal>03High [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Create mediawiki-core-qunit-selenium-composer-jessie - https://phabricator.wikimedia.org/T180482#3831295 (10zeljkofilipin) p:05Triage>03Normal [15:34:14] $log deploy zuul for parameter_functions update [15:34:19] !log deploy zuul for parameter_functions update [15:34:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:35:58] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831316 (10akosiaris) Pardon me, but I have to ask why a file with timestamps in the log file dating `Dec 11th`, and with a local... [15:38:22] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831321 (10awight) >>! In T181661#3831316, @akosiaris wrote: > Pardon me, but I have to ask why a file with timestamps in the log... [15:39:23] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831323 (10akosiaris) >>! In T181661#3831321, @awight wrote: >>>! In T181661#3831316, @akosiaris wrote: >> Pardon me, but I have t... [16:00:52] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831411 (10thcipriani) >>! In T181661#3831316, @akosiaris wrote: > Now for the more interesting stuff. I 've tried to run `/usr/bi... [16:20:28] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Castor: mediawiki-core-qunit-jessie node_modules cache ineffective - https://phabricator.wikimedia.org/T159591#3831492 (10debt) [16:20:31] 10Continuous-Integration-Config, 10Discovery, 10Discovery-Portal-Backlog, 10Wikimedia-Portals: CI tests on wikimedia/portals repo: cache node_modules to save time - https://phabricator.wikimedia.org/T152386#3831489 (10debt) 05Open>03stalled Moving this to stalled and to the backlog board, as we can pic... [16:24:47] 10MediaWiki-Codesniffer: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546#3831501 (10Esanders) Sounds good to me. [16:27:10] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3831511 (10zeljkofilipin) Any repository would do as far as I am concerne... [16:38:19] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3831570 (10zeljkofilipin) Looks like this is it: [[ https://www.mediawiki... [16:41:16] (03PS1) 10Chad: Removing git-logs. Doesn't work and dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/397864 [16:41:18] (03CR) 10Chad: [C: 032] Removing git-logs. Doesn't work and dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/397864 (owner: 10Chad) [16:41:49] (03Merged) 10jenkins-bot: Removing git-logs. Doesn't work and dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/397864 (owner: 10Chad) [16:56:00] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831602 (10akosiaris) `akosiaris@tin:/srv/deployment/ores/deploy$ scap deploy -v -l 'ores1004.eqiad.wmnet' T181661` fails reprodu... [17:13:42] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831654 (10akosiaris) >>! In T181661#3831411, @thcipriani wrote: >>>! In T181661#3831316, @akosiaris wrote: >> Now for the more in... [17:17:13] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3831661 (10zeljkofilipin) Argh. Looks like I am doing something wrong. `a... [17:19:16] 10Release-Engineering-Team, 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3831667 (10akosiaris) Moving the `deploy-cache/cache` directory aside did solve the issue (partially?) and moved on until... ```... [17:54:37] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3831772 (10zeljkofilipin) a:03zeljkofilipin [18:04:29] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3831802 (10zeljkofilipin) @dcausse apologies for the slow reply. ๐ŸŒ I have... [18:17:20] (03CR) 10Dduvall: [C: 032] SSD: Switch Mathoid to service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/395570 (owner: 10Thcipriani) [18:17:28] (03CR) 10jerkins-bot: [V: 04-1] SSD: Switch Mathoid to service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/395570 (owner: 10Thcipriani) [18:18:55] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [18:21:24] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3831842 (10debt) [18:21:26] 10Release-Engineering-Team (Watching / External), 10CirrusSearch, 10Discovery, 10Discovery-Search, 10Patch-For-Review: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#3831839 (10debt) 05Open>03Resolved a:03debt This was fixed with the move. [18:22:45] (03PS2) 10Thcipriani: SSD: Switch Mathoid to service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/395570 [18:22:56] !log deployed mobileapps@5b8796d to BC [18:23:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:08] (03CR) 10Dduvall: [C: 032] SSD: Switch Mathoid to service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/395570 (owner: 10Thcipriani) [18:27:17] (03Merged) 10jenkins-bot: SSD: Switch Mathoid to service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/395570 (owner: 10Thcipriani) [18:29:49] marxarelli: cool, thanks, deploying [18:29:51] ! [18:30:13] thcipriani: np [18:33:50] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/395570/ [18:34:10] huh [18:34:37] marxarelli: achievement unlocked. [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:58:55] 10Gerrit: gerrit's ipv6 param is failing on labs - https://phabricator.wikimedia.org/T182705#3831937 (10Paladox) [18:59:34] 10Gerrit, 10Operations: gerrit's ipv6 param is failing on labs - https://phabricator.wikimedia.org/T182705#3831948 (10Paladox) [19:22:48] 10Gerrit, 10Operations: gerrit's ipv6 param is failing on labs - https://phabricator.wikimedia.org/T182705#3832004 (10Dzahn) We talked about this. The problem here is manifold. Some facts: - IPv6 doesn't work in labs, this is causing more and more workarounds but yea, blocked by nova networking afaict - Gerri... [19:45:11] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 10Language-2017-Oct-Dec: [betacluster] Special:Translate does not allow to translate article content - multiple "Uncaught SyntaxError: Unexpected token < in JSON at position 0" - https://phabricator.wikimedia.org/T180841#3832055 (10greg) [20:01:25] 10Gerrit, 10Operations, 10Patch-For-Review: gerrit's ipv6 param is failing on labs - https://phabricator.wikimedia.org/T182705#3832298 (10Paladox) 05Open>03Resolved [20:22:31] 10Continuous-Integration-Config, 10Patch-For-Review: Puppet failing with labs ci class applied - https://phabricator.wikimedia.org/T182642#3832352 (10Paladox) 05Open>03Resolved a:03hashar [20:34:30] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:51:12] 10Scap, 10MediaWiki-Platform-Team: 'sql' script doesn't support multi-instance hosts - https://phabricator.wikimedia.org/T182713#3832376 (10Anomie) [21:47:50] PROBLEM - Free space - all mounts on deployment-sca03 is CRITICAL: CRITICAL: deployment-prep.deployment-sca03.diskspace._srv.byte_percentfree (<100.00%) [22:05:22] Hmm, is there any reason that we can't do phpcs for non-whitelisted users? [22:20:33] 10Release-Engineering-Team, 10User-greg: Create #wikimedia-releng-feed and move bots there - https://phabricator.wikimedia.org/T181582#3832662 (10greg) If no one is actioning on them, then let's just disable them (not the jobs, but the spamming of our channel). [22:25:20] (03PS1) 10Greg Grossmeier: JJB: Remove browsertests-irc [integration/config] - 10https://gerrit.wikimedia.org/r/397971 (https://phabricator.wikimedia.org/T181582) [22:28:58] (03PS1) 10Chad: More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 [22:29:00] (03CR) 10Chad: [C: 032] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 (owner: 10Chad) [22:29:51] (03CR) 10jerkins-bot: [V: 04-1] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 (owner: 10Chad) [22:30:07] (03CR) 10jerkins-bot: [V: 04-1] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 (owner: 10Chad) [22:31:22] 10Release-Engineering-Team, 10Scap, 10Scoring-platform-team: Scap is unhappy about deploying from a branch other than master - https://phabricator.wikimedia.org/T182498#3832681 (10greg) [22:33:43] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Wikidata: Make DataTypes repository read-only - https://phabricator.wikimedia.org/T182566#3832689 (10greg) [22:35:18] (03PS1) 10Chad: Fix typo from previous commit, and clean up branching logic [tools/release] - 10https://gerrit.wikimedia.org/r/397975 [22:35:20] (03CR) 10Chad: [C: 032] Fix typo from previous commit, and clean up branching logic [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [22:35:46] (03CR) 10jerkins-bot: [V: 04-1] Fix typo from previous commit, and clean up branching logic [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [22:35:50] (03CR) 10jerkins-bot: [V: 04-1] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 (owner: 10Chad) [22:35:53] (03CR) 10jerkins-bot: [V: 04-1] Fix typo from previous commit, and clean up branching logic [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [22:40:19] 10Release-Engineering-Team (Watching / External), 10Deployments, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3832696 (10greg) [23:30:57] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:36:10] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:26] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:45:26] (03Abandoned) 10Chad: More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397972 (owner: 10Chad) [23:45:37] (03PS2) 10Chad: More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397975 [23:45:45] PROBLEM - Puppet errors on integration-slave-k8s-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:46:05] (03CR) 10jerkins-bot: [V: 04-1] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [23:47:11] (03Abandoned) 10Chad: Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/397738 (owner: 10SaikrishnaS) [23:52:14] (03PS3) 10Chad: More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397975 [23:52:24] PROBLEM - Host integration-slave-k8s-1001 is DOWN: CRITICAL - Host Unreachable (10.68.23.217) [23:53:10] (03CR) 10Chad: [C: 032] More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [23:53:48] (03Merged) 10jenkins-bot: More descriptive failure [tools/release] - 10https://gerrit.wikimedia.org/r/397975 (owner: 10Chad) [23:54:13] 10Release-Engineering-Team (Kanban): Investigate the Extension Submittal Review process - https://phabricator.wikimedia.org/T182731#3832907 (10Jrbranaa)