[00:20:07] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:22:55] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:59:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (phase-out-trusty), 06Release-Engineering-Team: Switch MediaWiki coverage job from Trusty/Zend PHP 5.5 to Jessie/Zend PHP 7.0 - https://phabricator.wikimedia.org/T147778#3165659 (10Krinkle) [04:01:05] 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506#3165660 (10Krinkle) [05:28:15] I think there's an issue deleting jessie nodes. [05:28:18] on nodepool [06:40:57] Yippee, build fixed! [06:40:58] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #324: 09FIXED in 2 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/324/ [08:17:17] (03PS3) 10Umherirrender: [Pickle] Add Scribunto as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/344638 [12:18:50] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#3165966 (10Paladox) I think the new maintainer of trilead-ssh2 for jenkins will try and see if we can upgrade... [13:54:57] 10Continuous-Integration-Infrastructure, 07Tracking: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#3165993 (10Hydriz) [14:42:04] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3166076 (10Rammanojpotla) @hashar Is this issue is about adding the License text below footer and link it to README.md to the above two repos... [15:43:22] Yippee, build fixed! [15:43:23] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #384: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/384/ [15:51:35] Yippee, build fixed! [15:51:36] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #384: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/384/ [16:25:37] hashar nodepool looks down [16:25:38] https://integration.wikimedia.org/zuul/ [16:25:46] also hi [16:27:15] hashar: i can confirm ive been watching zuul and its been queueing but no progress [16:27:46] I belive legoktm reported on nodepool not deleting correctly early this morning [16:28:17] paladox: its been working fine until just now (ive noticed) i was up at around 6am utc-5 and it was operating normally [16:28:49] ok [16:29:16] paladox: it appears to fix itself [16:29:31] Yep, im guessing hashar fixed it :) [16:30:26] though it seems very slow [16:30:53] paladox: some of the test pipeline was already merged via gate n submit but for some reason zuul doesnt care [16:31:04] for exmaple all the mw-skins [16:31:07] ok [16:33:24] paladox: Zppix: hello [16:33:31] hi [16:33:35] yeah looks like there are 15 instance stuck in delete state [16:33:55] yep [16:36:26] o/ [16:36:37] I wonder if it getting stuck in deleting state is a bug. [16:40:02] https://review.openstack.org/#/c/55444/ [16:51:35] filling a task with some details [16:51:40] and will poke by sms the labs people [16:52:48] hashar: i perfer poking with a metal tong but thats me [16:53:07] ok [16:53:13] hashar thanks :) [16:58:56] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166228 (10hashar) [16:59:46] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166241 (10Paladox) p:05Triage>03High [17:23:47] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:48:45] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166249 (10chasemp) ```d35ae797-aa11-4a47-8236-e42e24c96abd dd4a2779-b256-4d71-9059-d7b65380f3d0 de6b9ef9-e531-4567-8090-621c35176b9d d6c32ccb-b68b-4... [17:51:37] hashar just wanted to say that the new maintainer has requested to be allowed to make releases for trilead-ssh2 which they said he will be granted by the end of weekend. He is planning to hopefully release a new release next week with the hmac256 support :). But i have tested using the trilead-ssh2 dep in the ssh slaves plugin and masking cores trilead class. But that causes side afects. So the only way it will work for now is to upgrade it [17:51:38] in jenkins. [17:52:46] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166250 (10Zppix) a:03chasemp Thanks Chase! [17:53:33] paladox: great... [17:54:30] yep. But that would require a jenkins update from 1.x to 2.x and i am unsure weather they will also backport it to one of the current lts release. As jenkins now requires java 8 for the next lts release. [17:55:29] Zppix also great news. We are going all the to support the new current ssh standards. [17:55:37] So jenkins is getting alot more secure [17:55:48] its about time [17:56:00] trilead has not been updated since 2014 [17:56:39] im shocked we've not done that ourselves by now [17:57:16] Shocked at what? [17:58:11] trilead not being updated for so long [17:58:37] ok [17:58:50] jenkins forked trilead-ssh2 [17:59:21] trilead-ssh2 original code is now un maintained. Though it seems google have some how forked it and have a bot that uses it now. [18:03:46] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:10] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166253 (10Zppix) 05Open>03Resolved Appears to be fixed according to integration.wikimedia.org/ci [18:06:28] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166255 (10Paladox) 05Resolved>03Open Should leave it open for investigation or until @chasemp or @hasher says it is good. [18:15:24] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3166259 (10Paladox) @hashar apparently zuul can ddos gerrit. This is fixed in https://github.com/openstack-infra/zuul/commit/12d9aa67e68a833cfbb3b7d5ecddd8c9f5e3fc5f [18:16:38] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3166261 (10Paladox) Also see this https://github.com/openstack-infra/zuul/commit/773651ad7bf0fc6adba2357173ffb657d874478a fix for depends on. [18:33:53] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166304 (10Andrew) 05Open>03Resolved This looks resolved to me. The main action item that I can think of is https://phabricator.wikimedia.org/T9... [18:38:58] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3166308 (10hashar) Indeed it is resolved. There are some components in OpenStack that end up being stuck from time to time, that is a known issue an... [19:33:47] (03PS2) 10Hashar: Add email of author/contributor [integration/config] - 10https://gerrit.wikimedia.org/r/346825 (owner: 10D3r1ck01) [19:35:46] (03CR) 10Hashar: [C: 032] Add email of author/contributor [integration/config] - 10https://gerrit.wikimedia.org/r/346825 (owner: 10D3r1ck01) [19:37:04] (03Merged) 10jenkins-bot: Add email of author/contributor [integration/config] - 10https://gerrit.wikimedia.org/r/346825 (owner: 10D3r1ck01) [19:37:26] (03PS4) 10Hashar: Add Wikimedia-Emoji-Bot repo for integration testing [integration/config] - 10https://gerrit.wikimedia.org/r/346819 (owner: 10D3r1ck01) [19:53:16] (03CR) 10Hashar: [C: 04-1] Add Wikimedia-Emoji-Bot repo for integration testing (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/346819 (owner: 10D3r1ck01) [19:53:24] (03PS5) 10D3r1ck01: Add Wikimedia-Emoji-Bot repo for integration testing [integration/config] - 10https://gerrit.wikimedia.org/r/346819 [19:54:26] (03CR) 10Hashar: [C: 032] "\O/" [integration/config] - 10https://gerrit.wikimedia.org/r/346819 (owner: 10D3r1ck01) [19:55:38] (03Merged) 10jenkins-bot: Add Wikimedia-Emoji-Bot repo for integration testing [integration/config] - 10https://gerrit.wikimedia.org/r/346819 (owner: 10D3r1ck01) [20:48:29] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]