[00:31:15] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:51:01] phab question: I can't find the edit form to set the "is sprint" flag and adjust start/end dates on a project. I thought this used to happen on the main "manage > edit details" form, but I'm not seeing those options on new projects (eg ) or ones that are already sprints (eg ) [03:40:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:15:43] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:20:42] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:51:30] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:21:29] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [06:00:38] (03PS1) 10Legoktm: Initial commit [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/412618 (https://phabricator.wikimedia.org/T71519) [06:01:27] (03CR) 10Legoktm: [V: 032 C: 032] Initial commit [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/412618 (https://phabricator.wikimedia.org/T71519) (owner: 10Legoktm) [06:55:41] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:58:00] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:28:01] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:39] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3982280 (10elukey) @mmodell let's remember that https://wikitech.wikimedia.org/wiki/Incident_doc... [07:51:24] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3982284 (10elukey) >>! In T182832#3981076, @mmodell wro... [09:12:31] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:42:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Proton, 10Readers-Web-Backlog, and 2 others: Set up Jenkins for chromium-render repository - https://phabricator.wikimedia.org/T179552#3982598 (10phuedx) Aside: That the above is a Phab keyword/meme, is as awesome as the hard work done... [09:47:31] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [10:03:38] !log deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions [10:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:08:17] Project beta-scap-eqiad build #196212: 04FAILURE in 4 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196212/ [10:11:49] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<30.00%) [10:16:08] 10Phabricator, 10WMNO-General: Herald rule for WMNO - https://phabricator.wikimedia.org/T187539#3982687 (10jhsoby-WMNO) Thanks @Aklapper! [10:16:55] Yippee, build fixed! [10:16:56] Project beta-scap-eqiad build #196213: 09FIXED in 3 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196213/ [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:53] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [12:15:00] (03PS2) 10Hashar: [Minifier] Remove extension jobs, archived [integration/config] - 10https://gerrit.wikimedia.org/r/410360 (https://phabricator.wikimedia.org/T187574) (owner: 10Jforrester) [12:15:22] (03CR) 10Hashar: [C: 032] "Attached to T187574" [integration/config] - 10https://gerrit.wikimedia.org/r/410360 (https://phabricator.wikimedia.org/T187574) (owner: 10Jforrester) [12:16:32] (03Merged) 10jenkins-bot: [Minifier] Remove extension jobs, archived [integration/config] - 10https://gerrit.wikimedia.org/r/410360 (https://phabricator.wikimedia.org/T187574) (owner: 10Jforrester) [12:21:20] (03CR) 10Hashar: [C: 032] "That is probably good. I need the containers published first then will add some experimental jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/411301 (owner: 10Hashar) [12:22:49] (03Merged) 10jenkins-bot: docker: containers for php5.6 based on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/411301 (owner: 10Hashar) [12:30:53] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:43:15] 10Phabricator: Integrate login with SUL - https://phabricator.wikimedia.org/T179124#3983215 (10Aklapper) 05stalled>03declined Unfortunately closing this report as no further information has been provided. @Kerry_Raymond: After you have provided information asked for and if this still happens, please set the... [12:43:52] (03PS1) 10Hashar: Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 [12:44:10] (03CR) 10Hashar: [C: 032] "They all pass!" [integration/config] - 10https://gerrit.wikimedia.org/r/412688 (owner: 10Hashar) [12:45:30] (03CR) 10jerkins-bot: [V: 04-1] Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 (owner: 10Hashar) [12:46:18] (03CR) 10jerkins-bot: [V: 04-1] Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 (owner: 10Hashar) [12:47:35] (03PS2) 10Hashar: Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 [12:47:48] (03CR) 10Hashar: [C: 032] Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 (owner: 10Hashar) [12:47:55] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#3983228 (10Aklapper) @WMDE-Fisch: Whoops, done now! [12:50:09] (03Merged) 10jenkins-bot: Migrate composer-php56 jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412688 (owner: 10Hashar) [12:55:15] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:44] (03PS1) 10Hashar: Clean up some composer jobs that ran on Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/412692 [13:08:56] (03CR) 10Hashar: [C: 032] Clean up some composer jobs that ran on Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/412692 (owner: 10Hashar) [13:11:04] (03Merged) 10jenkins-bot: Clean up some composer jobs that ran on Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/412692 (owner: 10Hashar) [14:03:02] 10Gerrit: Wrong RU-translation of createacct-benefit-body* - https://phabricator.wikimedia.org/T187724#3983467 (10Sergezolotukhin) [14:04:45] andre__ hi, see ^^. Im not sure what to change the project too. [14:05:01] but that looks like either mw core or an ext. [14:08:55] paladox: ? [14:09:04] how am I related here? [14:09:20] andre__ i thought you re tag projects if they are under the wrong tag? [14:10:01] paladox, I do. Is it that urgent that it requires pinging me? [14:10:08] plus other people do the same. [14:10:30] Nope. [14:11:29] paladox: then I don't understand why you ping me specifically. [14:11:38] ok [14:12:33] If you want to fix the tag, feel free to fix the tag. [14:12:52] or just remove #Gerrit if it's not about Gerrit and you're unsure what other project it should be. [14:12:58] No need to ping me specifically. [14:13:33] ok [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:59:07] and here we go with beta cluster puppet errors again :( [15:07:38] (03PS1) 10Hashar: Add hhvm-compile and php55-compile to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412718 [15:11:53] 10Release-Engineering-Team (Kanban): Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#3983680 (10zeljkofilipin) [15:12:09] Argument #2 (No Value) of PHPUnit_Framework_Assert::assertArrayHasKey() must be a array or ArrayAccess [15:12:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#3983690 (10zeljkofilipin) [15:12:16] anyone know what that means ? [15:12:33] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#3983680 (10zeljkofilipin) [15:12:48] from MWDebugTest::testAppendDebugInfoToApiResultXmlFormat [15:13:36] https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm-jessie/19600/console [15:15:52] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#3983701 (10zeljkofilipin) [15:19:19] ah, every patch is failing on this it seems. [15:21:25] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#3983728 (10zeljkofilipin) [15:24:57] hmm, it seems the phpunit test coverage wass expanded, cause this... [15:36:55] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#3983778 (10MarcoAurelio) [15:37:02] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#3983788 (10MarcoAurelio) p:05Triage>03High [15:56:26] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), and 4 others: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3983826 (10zeljkofilipin) @Ladsgroup this is resolved, right? I... [16:13:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), and 4 others: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3983847 (10Ladsgroup) 05Open>03Resolved yup [16:33:17] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:40:18] (03CR) 10Legoktm: [C: 032] Replace substr_compare with substr [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/411587 (https://phabricator.wikimedia.org/T187575) (owner: 10Umherirrender) [16:48:06] 10MediaWiki-Codesniffer: MissingCovers sniff should not warn on abstract classes - https://phabricator.wikimedia.org/T187738#3983886 (10Legoktm) [16:48:46] 10MediaWiki-Codesniffer: Add PHPCS rule to autofix some aliased PHP function names for consistency - https://phabricator.wikimedia.org/T183220#3983896 (10Legoktm) 05Open>03Resolved a:03Umherirrender I think this is fine for now - we can add more as they come up. [17:07:48] (03Merged) 10jenkins-bot: Replace substr_compare with substr [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/411587 (https://phabricator.wikimedia.org/T187575) (owner: 10Umherirrender) [17:12:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 3 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3983920 (10zeljkofilipin) >>! In T185928#3934433, @... [17:13:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 3 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3983922 (10zeljkofilipin) >>! In T185928#3934433, @... [17:13:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3983923 (10zeljkofilipin) [17:22:08] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:22:44] (03CR) 10jenkins-bot: Replace substr_compare with substr [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/411587 (https://phabricator.wikimedia.org/T187575) (owner: 10Umherirrender) [17:24:00] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:41:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul: Zuul no longer starts gate-and-submit when parent is in gate-and-submit pipeline - https://phabricator.wikimedia.org/T186949#3983943 (10Lucas_Werkmeister_WMDE) I’m still experiencing the same behavior of changes further up... [18:21:36] 10Continuous-Integration-Infrastructure: mediawiki-phpunit-hhvm-jessie jobs are always failing - https://phabricator.wikimedia.org/T187740#3984033 (10matmarex) [18:23:16] (03PS2) 10Hashar: Add hhvm-compile and php55-compile to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412718 [18:30:25] DockerIsA_Sin: if I need to revert a integration/jenkins change, do I also need to refresh the nodepool images or will it get picked up automatically? [18:32:00] (03PS1) 10Legoktm: Revert "mwconf: Use built-in DevelopmentSettings.php instead when available" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412753 (https://phabricator.wikimedia.org/T187581) [18:34:12] legoktm: that will be picked up everyday at 14:14 UTC [18:34:18] legoktm: or we can force an update [18:34:20] hashar lol [18:34:27] hashar: we need to force an update [18:34:37] (03CR) 10Legoktm: [C: 032] Revert "mwconf: Use built-in DevelopmentSettings.php instead when available" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412753 (https://phabricator.wikimedia.org/T187581) (owner: 10Legoktm) [18:34:45] legoktm: will launch it [18:34:45] how can I do that? [18:34:46] https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot :) [18:34:53] ah [18:34:55] right [18:34:59] not sure whether you have access to labnodepool1001.eqiad.wmnet [18:35:18] (03Merged) 10jenkins-bot: Revert "mwconf: Use built-in DevelopmentSettings.php instead when available" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412753 (https://phabricator.wikimedia.org/T187581) (owner: 10Legoktm) [18:35:24] I do! [18:35:39] should I give it a try then? [18:35:42] so what the wiki says [18:35:44] become-nodepool [18:35:52] (which really just use the proper sudo to login as nodepool user) [18:36:01] then refresh the local git repo [18:36:06] and then it asks to rebuild the snapshot [18:36:09] that takes a while [18:36:14] (15-20 mins) [18:36:14] should I do it in a screen? [18:36:18] probably [18:36:33] I think if you term get killed, it will continue in the background [18:36:35] but yeah [18:36:39] a screen will not hurt [18:36:53] !log manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie) [18:36:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:37:15] does it automatically delete instances based off of the old image once the new one is ready? [18:37:32] nop [18:37:40] the instances still around get consumed by the jobs [18:37:47] and new ones will spawned with the new snapshot [18:37:56] though one can delete the instances one by one via: nodepool delete 123456 [18:38:05] (where 123456 is the id of the instance to delete) [18:38:14] that flag them for deletion [18:38:20] and the daemon will delete them eventually [18:38:38] 2018-02-19 18:38:30,654 INFO paramiko.transport: Connected (version 2.0, client OpenSSH_6.7p1) [18:38:38] 2018-02-19 18:38:30,818 INFO paramiko.transport: Authentication (publickey) failed. [18:38:38] 2018-02-19 18:38:30,986 INFO paramiko.transport: Authentication (publickey) failed. [18:38:38] 2018-02-19 18:38:30,987 INFO nodepool.utils: Password auth exception. Try number 1... [18:39:03] it's in a loop doing that [18:39:15] oh, now it's working... [18:39:18] weird [18:40:47] legoktm: na that is normal [18:40:50] just give it some time :] [18:41:25] I will be back later [18:41:34] alright :) [18:41:34] kids && story time && dinner [18:48:25] 2018-02-19 18:46:50,396 INFO nodepool.SnapshotImageUpdater: Image snapshot-ci-jessie-1519065400 in wmflabs-eqiad is ready [18:52:38] !log deleted all current nodepool instances for ci-jessie [18:52:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:10:22] 10Continuous-Integration-Config, 10Zuul: mediawiki-phpunit-hhvm-jessie jobs are always failing - https://phabricator.wikimedia.org/T187740#3984129 (10matmarex) It seems fixed now. Build #19617 (Feb 19, 2018 6:57:56 PM) has just passed. Who do we thank for that? [19:10:43] legoktm: ^^ looks like you have fixed it :) [19:12:15] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#3984131 (10MarcoAurelio) `PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above t... [19:27:54] (03PS3) 10Hashar: Add hhvm-compile and php55-compile to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412718 [19:28:22] (03CR) 10Hashar: [C: 04-2] "luasandbox and wikidiff2 slightly differ :^\" [integration/config] - 10https://gerrit.wikimedia.org/r/412718 (owner: 10Hashar) [19:38:20] 10Gerrit, 10MediaWiki-Vagrant, 10Patch-For-Review: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#3984190 (10Tgr) 05Open>03Resolved a:03Tgr Haven't encountered this lately, let's call it fixed. [19:49:12] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #115: 04FAILURE in 2 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/115/ [20:57:27] (03PS4) 10Hashar: Add hhvm-compile and php55-compile to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412718 [21:08:21] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:22:48] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#3984337 (10MarcoAurelio) `PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical thres... [21:33:04] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#3983778 (10Paladox) I think this was just deleted but was never removed from shinken. [21:35:18] (03Abandoned) 10MarcoAurelio: zuul: archive the Wiki Object Model extension [integration/config] - 10https://gerrit.wikimedia.org/r/409851 (https://phabricator.wikimedia.org/T186516) (owner: 10MarcoAurelio) [21:36:40] can someone just abandon https://gerrit.wikimedia.org/r/#/c/369869/ ? [21:36:50] the owner of the patch is globally banned now [21:45:52] (03PS1) 10MarcoAurelio: zuul: archive Semantic Page Maker and Wiki Object Model [integration/config] - 10https://gerrit.wikimedia.org/r/412815 (https://phabricator.wikimedia.org/T187310) [21:46:01] Project mwext-phpunit-coverage-publish build #1229: 04FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1229/ [21:46:29] Yippee, build fixed! [21:46:30] Project mwext-phpunit-coverage-publish build #1230: 09FIXED in 28 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1230/ [21:55:50] (03PS5) 10Hashar: Add hhvm-compile and php55-compile to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/412718 [21:55:52] (03PS1) 10Hashar: Docker: hhvm container based on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/412819 [21:55:59] pass! :) [21:56:03] good night [22:00:24] 10Diffusion, 10MediaWiki-extensions-CentralNotice, 10Repository-Admins: Diffusion link for CentralNotice on Special:Version doesn't work - https://phabricator.wikimedia.org/T187077#3963593 (10MarcoAurelio) @Pcoombe It works for me in the current Special:Version link for meta.wikimedia. Perhaps we should make... [22:03:43] 10Diffusion, 10MediaWiki-extensions-CentralNotice, 10Repository-Admins: Diffusion link for CentralNotice on Special:Version doesn't work - https://phabricator.wikimedia.org/T187077#3984394 (10MarcoAurelio) Current version displayed at Meta is https://phabricator.wikimedia.org/rECNOa311619220c851c2a5d1fac0933... [22:04:39] 10Continuous-Integration-Config, 10Zuul: mediawiki-phpunit-hhvm-jessie jobs are always failing - https://phabricator.wikimedia.org/T187740#3984399 (10Legoktm) 05Open>03Resolved a:03Legoktm Me :-) It was what I had suspected in T187581 - the change to DevelopmentSettings.php. I reverted the change and reb... [22:06:50] 10Diffusion, 10MediaWiki-extensions-CentralNotice, 10Repository-Admins: Diffusion link for CentralNotice on Special:Version doesn't work - https://phabricator.wikimedia.org/T187077#3963593 (10Legoktm) Will this be fixed once {95f9a9f9014242a8a7e18ebdef0afdf8d16b1e9f} is deployed? [22:09:53] 10Diffusion, 10MediaWiki-extensions-CentralNotice, 10Repository-Admins: Diffusion link for CentralNotice on Special:Version doesn't work - https://phabricator.wikimedia.org/T187077#3984411 (10MarcoAurelio) I don't know t.b.h; but I'd not rush in enabling that. Gitiles is a bit rough compared to Diffusion IMHO. [22:19:39] (03CR) 10MarcoAurelio: "https://gerrit.wikimedia.org/r/#/c/412815/ takes care of both repos." [integration/config] - 10https://gerrit.wikimedia.org/r/409851 (https://phabricator.wikimedia.org/T186516) (owner: 10MarcoAurelio) [22:20:17] no_justification (it said i coulden't pm you so writing it here) Hi, question did you set "Zoranzoki21" as inactive? (if so) i can remove him from the git/reviewers page :) [22:21:12] paladox: the block was temporary afaik [22:21:21] Hauskatze oh. [22:21:25] Yes. Temporarily. Don't change the wiki page, he'll be unblocked soon [22:21:26] I will have to remove him [22:21:34] as it's breaking https://tools.wmflabs.org/gerrit-reviewer-bot/ [22:21:41] See if he behaves [22:21:53] "error: could not add Zoranzoki21: Account of Zoranzoki21 is inactive." [22:22:14] hmm ok, i will leave it for now [22:25:29] with my grumpy-old-man-hat on... I think he'll become 'inactive' soon [22:38:37] (03CR) 10Hashar: [C: 032] Docker: hhvm container based on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/412819 (owner: 10Hashar) [22:40:32] (03Merged) 10jenkins-bot: Docker: hhvm container based on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/412819 (owner: 10Hashar) [22:54:02] (03PS1) 10Krinkle: mwconf: Re-enable use of built-in DevelopmentSettings.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412825 (https://phabricator.wikimedia.org/T187581) [23:32:46] (03CR) 10Krinkle: [C: 032] mwconf: Re-enable use of built-in DevelopmentSettings.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412825 (https://phabricator.wikimedia.org/T187581) (owner: 10Krinkle) [23:33:32] (03Merged) 10jenkins-bot: mwconf: Re-enable use of built-in DevelopmentSettings.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/412825 (https://phabricator.wikimedia.org/T187581) (owner: 10Krinkle) [23:35:58] ok [23:36:00] legoktm: Looks like the pull command brought in unrelated changes [23:36:09] I assume that's what hashar is working on [23:37:21] Only affects files in */hhvm-jessie/* so probably won't affect the image I want to rebuild, is that right? [23:37:55] it's in dockerfiles/ so it doesn't matter at all [23:38:32] OK [23:38:41] So snapshot-ci-jessie is the one I want to rebuild? [23:38:44] The command as-is? [23:39:27] yes [23:40:44] !log Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825 [23:40:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:42:17] Got `/usr/lib/python2.7/dist-packages/paramiko/client.py:580: UserWarning: Unknown ssh-rsa host key for ..: ...` at some point. [23:42:25] a "future" warning [23:54:50] Krinkle: yeah, you can ignore all of those [23:54:54] and the key error ones too [23:55:04] K. it finished [23:55:18] now you have to go through `nodepool list` and delete all the images one by one [23:55:21] `nodepool delete id` [23:56:53] legoktm: Hm.. not sure I understand. [23:57:00] This is a list of standby images? [23:57:10] How do I know which ones are from before and which ones not? [23:58:29] I just deleted them all [23:58:42] Probably one or two false positives