[02:24:34] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[03:11:43] <wmf-insecte>	 Project mediawiki-core-code-coverage build #3002: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3002/
[03:12:49] <legoktm>	 Trying to @cover or @use not existing method "Sanitizer::urlEscapeId"
[03:14:12] <legoktm>	 commented on https://gerrit.wikimedia.org/r/375099
[03:34:06] <wikibugs>	 (03PS1) 10KartikMistry: Add apertium-crh, -tur and -crh-tur packages [integration/config] - 10https://gerrit.wikimedia.org/r/377391 (https://phabricator.wikimedia.org/T174765)
[04:46:48] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[04:56:49] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[05:36:48] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[07:25:34] <wikibugs>	 (03PS1) 10KartikMistry: Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987)
[07:36:40] <wikibugs>	 (03CR) 10Hashar: "In composer.json, you might want to add:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/377042 (owner: 10Legoktm)
[07:48:35] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Provision Docker >= 17.05 on contint1001 - https://phabricator.wikimedia.org/T175293#3599497 (10hashar) In apt.wikimedia.org we have:  | docker.io | 1.6.2~dfsg1-1~bpo8+1 | http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 P...
[07:53:54] <wikibugs>	 10Release-Engineering-Team (Next), 10Release Pipeline: Secret storage on contint1001 for Docker registry password - https://phabricator.wikimedia.org/T175298#3599501 (10hashar) We have bunch of credentials in https://integration.wikimedia.org/ci/credentials/ ,  they can then be exported as environment variable...
[07:55:20] <Amir1>	 hashar: Bonjour, Is scap okay today? Can I add things to SWAT?
[07:55:29] <Amir1>	 (If not, how can I help?)
[07:55:35] <hashar>	 I have no clue
[07:55:47] <hashar>	 potentially yeah should have been fixed. It seems some stuff got deployed yesterday
[07:56:42] <Amir1>	 okay, let me add those :)
[07:58:30] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add apertium-crh, -tur and -crh-tur packages [integration/config] - 10https://gerrit.wikimedia.org/r/377391 (https://phabricator.wikimedia.org/T174765) (owner: 10KartikMistry)
[07:58:36] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987) (owner: 10KartikMistry)
[07:59:29] <wikibugs>	 (03Merged) 10jenkins-bot: Add apertium-crh, -tur and -crh-tur packages [integration/config] - 10https://gerrit.wikimedia.org/r/377391 (https://phabricator.wikimedia.org/T174765) (owner: 10KartikMistry)
[07:59:31] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987) (owner: 10KartikMistry)
[08:00:30] <wikibugs>	 (03PS2) 10Hashar: Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987) (owner: 10KartikMistry)
[08:02:00] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987) (owner: 10KartikMistry)
[08:02:56] <wikibugs>	 (03Merged) 10jenkins-bot: Add apertium-cat-srd [integration/config] - 10https://gerrit.wikimedia.org/r/377408 (https://phabricator.wikimedia.org/T174987) (owner: 10KartikMistry)
[08:29:54] <wikibugs>	 10Gerrit, 10Developer-Relations, 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3599535 (10Qgil) p:05Triage>03Low Let me insist.  :)  >>! In T161901#3196709, @Qgil wrote: > Is https://www.mediawiki.org/wiki/Gerrit/Getti...
[08:35:10] <wikibugs>	 10Beta-Cluster-Infrastructure: Deployment wiki is flooded by spam and should be cleaned up, perhaps even restricted more - https://phabricator.wikimedia.org/T175197#3585793 (10Sau226) Mainframe do you need or want steward rights across the entire network so you can stop spam cross-wiki? Please tell me your opini...
[08:51:52] <shinken-wm>	 RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0]
[09:30:13] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Package Blubber - https://phabricator.wikimedia.org/T175609#3599700 (10Joe) `dh-make-golang` is what I'd use for creating a debian package from scratch, as it will also prepare packages for any dependency (read: any library dependency that stil...
[09:44:21] <wikibugs>	 (03PS6) 10MarcoAurelio: Whitelist Dvorapa on Zuul CI [integration/config] - 10https://gerrit.wikimedia.org/r/375765
[09:44:37] <wikibugs>	 (03CR) 10MarcoAurelio: "@hashar: Got your approval? Regards." [integration/config] - 10https://gerrit.wikimedia.org/r/375765 (owner: 10MarcoAurelio)
[10:46:23] <wikibugs>	 (03PS1) 10Hashar: dib: migrate out of <service>::packages [integration/config] - 10https://gerrit.wikimedia.org/r/377424
[10:47:12] <wikibugs>	 (03CR) 10Hashar: [C: 032] "graphoid::packages , mathoid::packages, trendingedits::packages all disappeared in favor of adding the packages directly in the profile." [integration/config] - 10https://gerrit.wikimedia.org/r/377424 (owner: 10Hashar)
[10:48:06] <wikibugs>	 (03Merged) 10jenkins-bot: dib: migrate out of <service>::packages [integration/config] - 10https://gerrit.wikimedia.org/r/377424 (owner: 10Hashar)
[10:48:56] <hashar>	 !log nodepool: force updating jessie image to grab php5.5-luasandbox - T161882 T174972
[10:49:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:49:02] <stashbot>	 T161882: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882
[10:49:02] <stashbot>	 T174972: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972
[10:50:51] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3578458 (10hashar) And I have packaged php-luasandbox. The bulk of the work is done, what is left is mayb...
[11:09:04] <hashar>	 !log Image snapshot-ci-jessie-1505213295 in wmflabs-eqiad is ready
[11:09:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:26:47] <wikibugs>	 (03PS1) 10Aude: Bump Wikidata to wmf/1.30.0-wmf.18 [tools/release] - 10https://gerrit.wikimedia.org/r/377431
[11:27:13] <wikibugs>	 (03CR) 10Aude: [C: 032] Bump Wikidata to wmf/1.30.0-wmf.18 [tools/release] - 10https://gerrit.wikimedia.org/r/377431 (owner: 10Aude)
[11:30:05] <wikibugs>	 (03Merged) 10jenkins-bot: Bump Wikidata to wmf/1.30.0-wmf.18 [tools/release] - 10https://gerrit.wikimedia.org/r/377431 (owner: 10Aude)
[11:55:35] <wikibugs>	 (03CR) 10Phedenskog: Run WebPageTest tests from Asia to verify the new cache pop. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/362972 (https://phabricator.wikimedia.org/T168416) (owner: 10Phedenskog)
[12:08:13] <wikibugs>	 (03PS5) 10Phedenskog: Run WebPageTest tests from Asia to verify the new cache pop. [integration/config] - 10https://gerrit.wikimedia.org/r/362972 (https://phabricator.wikimedia.org/T168416)
[12:09:58] <wikibugs>	 (03PS6) 10Phedenskog: Run WebPageTest tests from Asia to verify the new cache pop. [integration/config] - 10https://gerrit.wikimedia.org/r/362972 (https://phabricator.wikimedia.org/T168416)
[12:32:33] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate mwext-textextension-php55 to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377448 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[12:33:33] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate mwext-textextension-php55 to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377448 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[12:39:15] <wikibugs>	 (03PS1) 10Hashar: Strip '-jessie' prefix from a Zuul template [integration/config] - 10https://gerrit.wikimedia.org/r/377450
[12:43:51] <wikibugs>	 (03CR) 10Hashar: [C: 032] "Noop in Zuul :]" [integration/config] - 10https://gerrit.wikimedia.org/r/377450 (owner: 10Hashar)
[12:44:04] <wikibugs>	 (03PS1) 10Hashar: mwext-testextension-php55 to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377452 (https://phabricator.wikimedia.org/T161882)
[12:45:41] <wikibugs>	 (03Merged) 10jenkins-bot: Strip '-jessie' prefix from a Zuul template [integration/config] - 10https://gerrit.wikimedia.org/r/377450 (owner: 10Hashar)
[12:46:29] <wikibugs>	 (03CR) 10Hashar: [C: 032] mwext-testextension-php55 to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377452 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[12:47:26] <wikibugs>	 (03Merged) 10jenkins-bot: mwext-testextension-php55 to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377452 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[12:51:33] <wikibugs>	 (03PS1) 10Hashar: Remove some trusty jobs [integration/config] - 10https://gerrit.wikimedia.org/r/377453 (https://phabricator.wikimedia.org/T161882)
[12:52:17] <Krinkle>	 hashar: IS there a task for the infra/jenkins issue? Is that also the reason for no wmf branch this week?
[12:52:29] <Krinkle>	 (per wikitech:Deployments)
[12:52:43] <hashar>	 Krinkle: which issue?
[12:53:25] <hashar>	 hoaoo
[12:53:29] <Krinkle>	 hashar: Deployments cancelled due "to Jenkins/infra issue"
[12:53:40] <hashar>	 scap was not able to ssh to the mediawiki app servers due to some ssh config
[12:53:46] <hashar>	 but that one should be solved
[12:53:47] <hashar>	 solve
[12:53:52] <Krinkle>	 Okay
[12:53:58] <Krinkle>	 So wmf.18 will  still happen?
[12:54:02] <hashar>	 for the eevening swat, I have no diea
[12:54:07] <hashar>	 yeah wmf.18 should happen
[12:54:22] <hashar>	 we cut the branch directly in Gerrit 
[12:54:53] <Krinkle>	 Oh, right. Tuesday hasn't happened yet in SF
[12:54:59] <Krinkle>	 Okay, I misread
[12:55:00] <Krinkle>	 Thanks :)
[12:56:11] <tabbycat>	 Krinkle: I made a task for that and greg-g marked it as resolved
[12:56:37] <tabbycat>	 btw Krinkle now that you're here, I wonder if you could review some centralauth patches (2) ?
[12:59:20] <sau226>	 Lua error in package.lua at line 80: module 'Module:IPAc-en/data' not found. <<error in beta cluster wiki
[13:00:17] <wikibugs>	 (03PS1) 10Hashar: mwext-Wikibase-client-tests-mysql-php55 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377457 (https://phabricator.wikimedia.org/T161882)
[13:01:29] <wikibugs>	 (03CR) 10Hashar: [C: 04-2] "Pending tests on https://gerrit.wikimedia.org/r/#/c/70395/19" [integration/config] - 10https://gerrit.wikimedia.org/r/377457 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[13:09:28] <hashar>	 !log nodepool: deleting alien instance: openstack server delete ci-jessie-wikimedia-815477
[13:09:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:11:35] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10monitoring, 10Tracking, 10Wikimedia-Incident: Tracking: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#3600174 (10Gilles)
[13:44:39] <wikibugs>	 (03CR) 10Hashar: [C: 032] "Tested and each flavor has:" [integration/config] - 10https://gerrit.wikimedia.org/r/377457 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[13:44:47] <wikibugs>	 (03CR) 10Hashar: [C: 032] Remove some trusty jobs [integration/config] - 10https://gerrit.wikimedia.org/r/377453 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[13:46:42] <wikibugs>	 (03Merged) 10jenkins-bot: Remove some trusty jobs [integration/config] - 10https://gerrit.wikimedia.org/r/377453 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[13:46:45] <wikibugs>	 (03Merged) 10jenkins-bot: mwext-Wikibase-client-tests-mysql-php55 to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377457 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[13:47:10] <wmf-insecte>	 Yippee, build fixed!
[13:47:11] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #520: 09FIXED in 3 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/520/
[13:52:10] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3600334 (10hashar) We no more have any jobs on nodepool Trusty instances  (label: ci-trusty-wikimedia) `\o/`
[13:56:33] <paladox>	 hashar :)
[13:56:42] <paladox>	 does that mean we can go to java 8 now? :)
[13:56:53] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[14:00:41] <hashar>	 paladox: there are still some jobs running on the permanent Trusty slaves
[14:00:47] <paladox>	 oh
[14:00:59] <Zppix>	 hashar:  any news on the issue we had with eu swat yesterday?
[14:01:38] <tabbycat>	 Zppix: yes, Pacific Electric didn't received WMF's paycheck so they cut the power supplies :P
[14:01:59] <Zppix>	 tabbycat:  i feel sarcasm
[14:02:21] <tabbycat>	 yes: the real issue was scap not being able to ssh to some nodes
[14:02:24] <paladox>	 Zppix i think they fixed that
[14:02:26] <tabbycat>	 at least on EU SWAT
[14:02:32] <paladox>	 was due to a puppet change for scap
[14:02:38] <tabbycat>	 on evening swat, I think jenkins had some issues
[14:02:46] <paladox>	 jenkins stalled
[14:02:54] <tabbycat>	 in any case, congrats to those who helped resolve the issue
[14:03:02] <Zppix>	 paladox:  so i can reschedule my patch?
[14:03:22] <paladox>	 Zppix i think so.
[14:03:47] <Zppix>	 ok morning swat it is
[14:04:38] <wikibugs>	 (03PS1) 10Hashar: Move some jobs from Trusty to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377467 (https://phabricator.wikimedia.org/T161882)
[14:05:18] <hashar>	 !log Deleting integration-slave-trusty-1004  - T161882
[14:05:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:05:22] <stashbot>	 T161882: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882
[14:07:47] <sau226>	 Lua error in package.lua at line 80: module 'Module:Category handler/data' not found.
[14:08:53] <wikibugs>	 (03CR) 10Hashar: [C: 032] Move some jobs from Trusty to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377467 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:09:50] <shinken-wm>	 PROBLEM - Host integration-slave-trusty-1004 is DOWN: CRITICAL - Host Unreachable (10.68.23.143)
[14:10:01] <wikibugs>	 (03Merged) 10jenkins-bot: Move some jobs from Trusty to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377467 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:13:13] <paladox>	 hashar i wonder can this https://github.com/wikimedia/integration-config/blob/ec07802d8bfd44294411cda543fcf1887339c372/jjb/php.yaml#L35 be moved to nodepool?
[14:22:30] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban): Remove trusty from dib and clean out puppet - https://phabricator.wikimedia.org/T175696#3600481 (10hashar)
[14:22:44] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban): Remove trusty from dib and clean out puppet - https://phabricator.wikimedia.org/T175696#3600497 (10hashar)
[14:22:47] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3600498 (10hashar)
[14:22:53] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban): Remove trusty from dib and clean out puppet - https://phabricator.wikimedia.org/T175696#3600481 (10hashar) a:03hashar
[14:23:49] <wikibugs>	 (03PS1) 10Hashar: dib: remove Ubuntu Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/377475 (https://phabricator.wikimedia.org/T175696)
[14:24:24] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Remove trusty from Nodepool and clean out puppet - https://phabricator.wikimedia.org/T175696#3600507 (10hashar)
[14:27:39] <wikibugs>	 (03CR) 10Hashar: [C: 032] dib: remove Ubuntu Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/377475 (https://phabricator.wikimedia.org/T175696) (owner: 10Hashar)
[14:30:29] <wikibugs>	 (03Merged) 10jenkins-bot: dib: remove Ubuntu Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/377475 (https://phabricator.wikimedia.org/T175696) (owner: 10Hashar)
[14:30:53] <wikibugs>	 (03PS1) 10Hashar: Migrate composer-validate jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377477 (https://phabricator.wikimedia.org/T161882)
[14:31:52] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
[14:33:51] <wmf-insecte>	 Yippee, build fixed!
[14:33:52] <wmf-insecte>	 Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #514: 09FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/514/
[14:34:15] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate composer-validate jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377477 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:35:23] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate composer-validate jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377477 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:38:45] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[14:39:09] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[14:39:19] <wikibugs>	 (03PS1) 10Hashar: integration-composer-check-php55 to Nodepool and Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377478 (https://phabricator.wikimedia.org/T161882)
[14:40:17] <wikibugs>	 (03CR) 10Hashar: [C: 032] integration-composer-check-php55 to Nodepool and Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377478 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:41:26] <wikibugs>	 (03Merged) 10jenkins-bot: integration-composer-check-php55 to Nodepool and Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377478 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[14:46:39] <hashar>	 !log provisionning integration-slave-jessie-1003 and integration-slave-jessie-1004 to move php55lint to them.  NOT READY YET - T161882
[14:46:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:46:45] <stashbot>	 T161882: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882
[14:47:24] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3600613 (10hashar) Still have to switch the jobs using the label phpflavor-php55  labs-tools-ZppixBot-php5...
[14:48:17] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3600615 (10hashar)
[14:48:49] <hashar>	 The last jobs to migrate are the php55lint ones:  labs-tools-ZppixBot-php55lint , mediawiki-core-php55lint , mwgate-php55lint , operations-mw-config-php55lint , php55lint
[14:49:03] <hashar>	 which potentially can later be moved to Docker :]
[14:51:12] <Zppix>	 you rang?
[14:51:15] <Zppix>	 oh nevermind
[14:57:47] <hashar>	 !log Deleted all left over jenkins jobs having ci-trusty-wikimedia label. - T161882
[14:57:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:57:51] <stashbot>	 T161882: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882
[15:00:09] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3600691 (10hashar) And I have deleted some left over jobs that had the `ci-trusty-wikimedia` label althoug...
[15:04:09] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:06:59] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:07:25] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:08:46] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:09:18] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Package php modules for Zend 5.5 on Jessie - https://phabricator.wikimedia.org/T174972#3600734 (10hashar) https://gerrit.wikimedia.org/r/#/c/377469/2/modules/aptrepo/files/distributions-wikime...
[15:15:12] <hasharAway>	 idle &
[15:16:21] <wmf-insecte>	 Project mediawiki-core-code-coverage build #3003: 04STILL FAILING in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3003/
[15:39:18] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[16:13:08] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:19:39] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[16:22:30] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[16:22:50] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[16:27:29] <ottomata>	 yarrrr
[16:27:33] <ottomata>	 deployment-puppetmaster02 not rebasing well
[16:27:48] <ottomata>	 error: could not apply 9974c0f79a... swift: use implicit /dev/swift prefix for swift devices
[16:27:49] <ottomata>	 Recorded preimage for 'modules/swift/manifests/init_device.pp'
[16:27:49] <ottomata>	 Could not pick 9974c0f79ab7c8a1474849f91a616d656cde859d
[16:27:49] <ottomata>	 Rebase failed! See error messages above.
[16:27:49] <ottomata>	 Reverting rebase attempt
[16:28:35] <ottomata>	 ah that's godog
[16:28:37] <ottomata>	 ^
[16:28:54] <ottomata>	 godog: , i think you have a patch applied in deployment prep that won't rebase anymore
[16:29:13] <godog>	 yeah I'll poke at it tomorrow, gehel was mentioning it too, sorry today I can't
[16:29:41] <ottomata>	 hm, ok, it means that pupetmaster in deployment prep can't update puppet, can get rid of it from there?
[16:30:20] <ottomata>	 godog: ^?
[16:30:45] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:30:46] <ottomata>	 un-cherry pick it?
[16:31:03] <shinken-wm>	 PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:32:46] <shinken-wm>	 PROBLEM - Puppet errors on deployment-eventlog02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[16:32:52] <godog>	 ottomata: I'm doing puppet swat now, you can un cherry-pick it, please watch ms-be and ms-fe if they break
[16:33:04] <ottomata>	 in deployment-prep?
[16:33:30] <shinken-wm>	 PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:33:42] <ottomata>	 these are me ^
[16:33:48] <ottomata>	 will fix asap, all the sudden a meeting started too
[16:33:48] <ottomata>	 ahhh
[16:34:38] <godog>	 yeah in deployment-prep ottomata 
[16:39:44] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:40:08] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[16:42:27] <shinken-wm>	 PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[16:44:03] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[16:45:21] <shinken-wm>	 PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[16:45:26] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[16:53:07] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
[17:14:19] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:25:50] <wikibugs>	 10Beta-Cluster-Infrastructure: Deployment wiki is flooded by spam and should be cleaned up, perhaps even restricted more - https://phabricator.wikimedia.org/T175197#3601220 (10Mainframe98) While it would be nice, I don't see an immediate need for it. Granted, I'm only looking at deployment wiki for now, so I don...
[17:30:19] <wikibugs>	 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3601241 (10greg) 05Open>03Resolved Merged, deployed, done.
[17:45:07] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:47:46] <shinken-wm>	 RECOVERY - Puppet errors on deployment-eventlog02 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:48:31] <shinken-wm>	 RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:49:03] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:49:39] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:49:45] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:50:22] <shinken-wm>	 RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:50:27] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:50:45] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:51:02] <shinken-wm>	 RECOVERY - Puppet errors on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:52:26] <shinken-wm>	 RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0]
[17:57:27] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:57:50] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:17:16] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%)
[18:21:01] <greg-g>	 uh oh
[18:21:10] <greg-g>	 I'm migrating locations, bbiab
[18:21:53] <chasemp>	 hasharAway: thcipriani fyi that trusty removal landed in nodepool now
[18:25:26] <thcipriani>	 chasemp: ah, cool, that should make nodepool's logic a bit simpler. Thanks!
[18:32:01] <paladox>	 lol animoji
[18:32:03] <paladox>	 woops
[18:32:06] <paladox>	 wrong place
[18:39:48] <ottomata>	 FYI godog, i rebased and resolved conflicts, hopefully correctly.
[18:43:27] <shinken-wm>	 PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[18:49:43] <hasharAway>	 !log nodepool: deleted image image-ci-trusty_old_20170804   Keeping image-ci-trusty just in case
[18:49:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:00:04] <wikibugs>	 10Deployment-Systems, 10Tracking: Trebuchet blockers for MediaWiki (tracking) - https://phabricator.wikimedia.org/T45338#3601699 (10demon)
[19:02:25] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka-jumbo-1 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:03:55] <wikibugs>	 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10Operations: Trebuchet targets for test/testrepo are out of date - https://phabricator.wikimedia.org/T149180#3601711 (10demon) 05Open>03declined Nobody cares anymore.
[19:04:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: Ensure /srv/deployment/integration/slave-scripts is latest master on deployment-tin - https://phabricator.wikimedia.org/T97324#3601714 (10demon) 05Open>03declined Nobody cares anymore.
[19:05:15] <wikibugs>	 10Scap (Scap3-Adoption-Phase1), 10Wikimedia-Wikimania-Scholarships, 10Patch-For-Review, 10User-bd808: Deploy scholarships with scap3 - https://phabricator.wikimedia.org/T129134#3601716 (10mmodell) {icon check color=green} Done?
[19:06:56] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:09:27] <wmf-insecte>	 Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #115: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/115/
[19:10:12] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[19:18:27] <shinken-wm>	 RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0]
[19:20:24] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[19:24:44] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[19:25:28] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[19:37:49] <paladox>	 no_justification, sorry for ping just wanted to say i think in gerrit 2.15, it will upgrade the mysql connector which i did as it does java 8 https://gerrit-review.googlesource.com/#/c/gerrit/+/107230/ to 6.x. which means we wont be able to use the debian repo.
[19:40:05] <hashar>	 !log hacked integration-slave-jessie hosts to ship them php5.5
[19:40:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:49:47] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:50:09] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:50:23] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:50:25] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:16:05] <hashar>	 k puppet is happy
[20:20:24] <no_justification>	 hashar: Happy might not be the right word. Puppet is never /happy/ :p
[20:20:33] <no_justification>	 Maybe just: "not currently raging in anger?"
[20:20:34] <no_justification>	 Hehehehe
[20:21:48] <wikibugs>	 (03PS1) 10Hashar: Migrate phplint jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377556 (https://phabricator.wikimedia.org/T161882)
[20:30:59] <wikibugs>	 (03PS2) 10Hashar: Migrate phplint jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377556 (https://phabricator.wikimedia.org/T161882)
[20:32:22] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate phplint jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377556 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[20:33:30] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate phplint jobs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/377556 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar)
[20:35:45] <hashar>	 !log pooling integration-slave-jessie-1003 and integration-slave-jessie-1004
[20:35:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:37:06] <paladox>	 i get everything to work with soy, then it gets stuck on the tests.
[20:37:22] <paladox>	 they should create phpunits for java, i really want to ignore some things.
[20:38:22] <hashar>	 I am a job away from phasing out trusty entirely :]
[20:38:31] <paladox>	 :)
[20:38:36] <paladox>	 what job are you left on?
[20:38:38] <paladox>	 :)
[20:38:45] <hashar>	 php-compile-php55
[20:38:55] <hashar>	 will deal with it tomorrow
[20:39:06] <paladox>	 ok
[20:40:30] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-trusty), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3602189 (10hashar) php-compile-php55 is the last job still on Trusty.
[20:41:12] <hashar>	 paladox: and the Nodepool trusty instances are gone :]
[20:41:20] <paladox>	 :)
[20:41:20] <paladox>	 :)
[20:41:27] <hashar>	 so as soon as I manage to migrate that last job, that means that every single CI slave will be running Jessie
[20:41:49] <paladox>	 hashar that means we can go onto java 8 and jenkins 2.60 soon :)
[20:42:24] <hashar>	 yeahhhh :]
[20:42:45] <hashar>	 all since you mentionned it now requires java8 which is not on trusty
[20:42:51] <hashar>	 that was a good incentive to phase out trusty
[20:43:01] <hashar>	 at the price of spending a couple days figuring out how to package php5.5  for jessie
[20:43:01] <paladox>	 yeh
[20:43:05] <hashar>	 which is more or less done
[20:43:08] <paladox>	 :)
[20:43:19] <hashar>	 the port is terribly dirty though
[20:43:23] <hashar>	 but the tests pass
[20:43:36] <hashar>	 I even found a bug in mediawiki/core :]
[20:43:56] <paladox>	 oh
[20:44:01] <hashar>	 https://gerrit.wikimedia.org/r/#/c/377232/
[20:44:15] <hashar>	 on some edge case an exception is thrown, which prevent some state from being restored
[20:44:18] <paladox>	 :)
[20:44:31] <hashar>	 that potentially cause later tests to fail
[20:44:32] <hashar>	 !
[20:44:44] <hashar>	 tldr
[20:44:46] <hashar>	 I got a fatal error
[20:47:17] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%)
[20:47:41] <hashar>	 thcipriani: deployment-tin, we should probably rebuild it with more disk space
[20:47:48] <hashar>	 twentyafterfour was hinting at that iirc
[20:48:07] <wikibugs>	 10Scap, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3602262 (10Nuria) 05Open>03Resolved
[20:48:31] <hashar>	 ah yeah
[20:48:32] <hashar>	 https://phabricator.wikimedia.org/T166492
[20:48:35] <hashar>	 m1.large
[20:48:48] <hashar>	 IF ONLY WE COULD JUST CHANGE THE FLAVOR
[20:50:11] <thcipriani>	 yeah, larger disk would be nicer on that machine. we're definitely not very careful about disk space on deployment hosts.
[20:50:35] <hashar>	 well
[20:50:51] <thcipriani>	 removed the /srv/mediawiki/.git dir on deployment-tin since it will be regenerated and it's not used in deployment.
[20:51:03] <hashar>	 a few TBytes of disk in RAID is cheaper than us trying to figure out a way to save disk? :D
[20:51:07] <thcipriani>	 so /srv is at 57% usage now
[20:51:16] <hashar>	 !!!
[20:51:53] <thcipriani>	 need to figure out how to manage that repo better before it's ready for prime time. We've got it turned off behind a feature flagged in production.
[21:02:15] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)
[21:36:17] <hashar>	 sleep &
[21:37:53] <Zppix>	 O/ hashar
[21:57:58] <ebernhardson>	 so fun thing ... building a new mwv box. Tried multiple times but each time it tries to clone core it dies. Ran it manually, same thing. it gets as far as "remote: Compressing objects:  99% (676275/676276)" before "error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.". If anyone has ideas before i blow the next hour figuring out what it's 
[21:58:04] <ebernhardson>	 doing ..
[22:08:43] <wikibugs>	 10Release-Engineering-Team (Kanban): wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending - https://phabricator.wikimedia.org/T173477#3602431 (10Jrbranaa) 05Open>03Resolved Hello @aaron,  Per our chat.  I'd like to get some more info from you if possible.  We could do it a...
[22:09:32] <wikibugs>	 10Release-Engineering-Team (Kanban): wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending - https://phabricator.wikimedia.org/T173477#3602438 (10Jrbranaa)
[22:24:59] <bd808>	 ebernhardson: there's an open bug about that. it's some form of git + RNG hell
[23:23:24] <wikibugs>	 10Continuous-Integration-Infrastructure: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3602645 (10Jdforrester-WMF)
[23:24:52] <wikibugs>	 10Continuous-Integration-Infrastructure: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3602659 (10Jdforrester-WMF) Also integration-slave-jessie-1002 in https://integration.wikimedia.org/ci/job/mwext-VisualEditor-jsduck/6494/console
[23:25:41] <wikibugs>	 10Continuous-Integration-Infrastructure: integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3602663 (10Jdforrester-WMF) Aha. Last successful pass was on integration-slave-*trusty*-1001 with https://integration.wikimedia.org/ci/job/mwext-VisualEditor-...