[00:01:37] 6Release-Engineering-Team, 6Operations: Update gerrit sshkey in role::ci::slave::labs when upgrade to Jessie happens - https://phabricator.wikimedia.org/T131903#2182462 (10madhuvishy) [00:01:52] 6Release-Engineering-Team, 6Operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#2182475 (10madhuvishy) [00:01:54] 6Release-Engineering-Team, 6Operations: Update gerrit sshkey in role::ci::slave::labs when upgrade to Jessie happens - https://phabricator.wikimedia.org/T131903#2182474 (10madhuvishy) [00:02:46] 6Release-Engineering-Team, 6Operations: Update gerrit sshkey in role::ci::slave::labs when upgrade to Jessie happens - https://phabricator.wikimedia.org/T131903#2182462 (10madhuvishy) [00:43:40] Project selenium-Flow » firefox,beta,Linux,,contintLabsSlave && UbuntuTrusty build #15: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,VERSION=,label=contintLabsSlave%20&&%20UbuntuTrusty/15/ [00:54:03] (03PS1) 10Paladox: [LoopFunctions] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/281862 [01:41:29] thcipriani|afk: the deal with the staging area is that you need to set up ssh keys in phabricator to push. [01:49:20] it's weird. I've never had ssh keys setup and I've never had a problem until I commandeered hasher's patch. I got it figured out WRT to pushing to staging. .deb building failed for whatever reason, but tests passed... [01:51:53] We should probably document that on the Differential page (staging/pushing/testing needing? ssh keys) [01:52:36] the GitHub/Phabricator steps for repo creation should be documented [01:52:58] I made a repo in Gerrit earlier and found I had to make the GitHub repo manually, no idea about Phab [03:16:04] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1037: 04FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1037/ [03:33:41] 6Release-Engineering-Team, 6Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2182662 (10madhuvishy) More progress! 6. Ran into some issues while adding the gerrit ssh key to /etc/ssh/ssh_known_hosts on the Jenkins slaves - they are d... [05:32:38] Project beta-scap-eqiad build #96963: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96963/ [05:37:30] Project beta-scap-eqiad build #96964: 04STILL FAILING in 2 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96964/ [05:42:23] Project beta-scap-eqiad build #96965: 04STILL FAILING in 2 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96965/ [05:48:02] Project beta-scap-eqiad build #96966: 04STILL FAILING in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96966/ [05:57:53] Project beta-scap-eqiad build #96967: 04STILL FAILING in 2 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96967/ [06:07:54] Project beta-scap-eqiad build #96968: 04STILL FAILING in 2 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96968/ [06:17:51] Project beta-scap-eqiad build #96969: 04STILL FAILING in 2 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96969/ [06:28:05] Project beta-scap-eqiad build #96970: 04STILL FAILING in 2 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96970/ [06:34:27] Yippee, build fixed! [06:34:27] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #71: 09FIXED in 7 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/71/ [06:37:53] Project beta-scap-eqiad build #96971: 04STILL FAILING in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96971/ [06:47:45] Project beta-scap-eqiad build #96972: 04STILL FAILING in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96972/ [06:49:03] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:57:40] Project beta-scap-eqiad build #96973: 04STILL FAILING in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96973/ [07:01:26] PROBLEM - Free space - all mounts on deployment-sentry2 is CRITICAL: CRITICAL: deployment-prep.deployment-sentry2.diskspace._var.byte_percentfree (<44.44%) [07:03:37] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:07:36] Project beta-scap-eqiad build #96974: 04STILL FAILING in 2 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96974/ [07:19:50] Yippee, build fixed! [07:19:50] Project beta-scap-eqiad build #96975: 09FIXED in 4 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/96975/ [07:24:04] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:35] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [08:54:49] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:54:49] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:03:46] (03PS1) 10Hashar: [integration/jenkins] use composer to run phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/281896 [09:05:13] (03CR) 10Hashar: [C: 032] [integration/jenkins] use composer to run phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/281896 (owner: 10Hashar) [09:05:28] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:06:11] (03Merged) 10jenkins-bot: [integration/jenkins] use composer to run phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/281896 (owner: 10Hashar) [09:06:37] (03CR) 10Hashar: "recheck" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/277508 (https://phabricator.wikimedia.org/T124447) (owner: 10Mobrovac) [09:06:38] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:07:46] (03CR) 10Hashar: "Validated on last merged change https://gerrit.wikimedia.org/r/#/c/277508/" [integration/config] - 10https://gerrit.wikimedia.org/r/281896 (owner: 10Hashar) [09:10:31] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:14:51] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:22:53] 10Continuous-Integration-Config: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2182921 (10hashar) [09:25:48] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:32:46] why is puppet so broken on those mw hosts? [09:34:59] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [09:37:20] (03PS1) 10Hashar: Add experimental composer-* jobs to a couple repos [integration/config] - 10https://gerrit.wikimedia.org/r/281905 (https://phabricator.wikimedia.org/T131917) [09:38:46] (03CR) 10Hashar: [C: 032] Add experimental composer-* jobs to a couple repos [integration/config] - 10https://gerrit.wikimedia.org/r/281905 (https://phabricator.wikimedia.org/T131917) (owner: 10Hashar) [09:39:12] Krenair: look at their /var/log/puppet.log ? :-) [09:39:22] must be some change that got merged in operations/puppet [09:39:34] potentially a conflict in the puppet master repo [09:39:35] (03Merged) 10jenkins-bot: Add experimental composer-* jobs to a couple repos [integration/config] - 10https://gerrit.wikimedia.org/r/281905 (https://phabricator.wikimedia.org/T131917) (owner: 10Hashar) [09:39:38] (due to autorebase) [09:41:39] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep [09:41:45] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: SQLite3::BusyException: database is locked: INSERT INTO "fact_values" ("created_at", "fact_name_id", "host_id", "updated_at", "value") VALUES (?, ?, ?, ?, ?) [09:41:59] * Krenair will be back in a few hours [09:53:41] Krenair: yeah there are multiple puppet master run in parallel and it doesn't deal with sqlite locks properly :( [09:53:49] that will self solve on next run [09:56:31] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2183026 (10hashar) [09:57:04] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2182921 (10hashar) [10:09:21] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2183039 (10hashar) [10:11:03] 10Continuous-Integration-Config, 10Wikimedia-IEG-grant-review, 10Wikimedia-Wikimania-Scholarships, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2182921 (10hashar) p:5Triage>3Normal I have proposed patches for the couple... [10:14:29] (03Abandoned) 10Addshore: Remove extension ZeroPortal from gate extensions [integration/config] - 10https://gerrit.wikimedia.org/r/281206 (owner: 10Addshore) [10:14:52] 10Continuous-Integration-Config, 10Wikimedia-IEG-grant-review, 10Wikimedia-Wikimania-Scholarships, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2183047 (10hashar) I am not proposing a CI configuration change at this point,... [10:17:20] (03PS1) 10Hashar: Flag phpunit job has legacy [integration/config] - 10https://gerrit.wikimedia.org/r/281912 (https://phabricator.wikimedia.org/T131917) [10:17:28] (03CR) 10Hashar: [C: 032] Flag phpunit job has legacy [integration/config] - 10https://gerrit.wikimedia.org/r/281912 (https://phabricator.wikimedia.org/T131917) (owner: 10Hashar) [10:18:54] (03Merged) 10jenkins-bot: Flag phpunit job has legacy [integration/config] - 10https://gerrit.wikimedia.org/r/281912 (https://phabricator.wikimedia.org/T131917) (owner: 10Hashar) [10:20:20] hashar: hey, Please check out my comment in https://phabricator.wikimedia.org/D186 [10:25:30] Amir1: ah that make sense :-) [10:26:49] thanks for the review, lots of other ones are coming [10:27:52] trying to land it [10:28:27] Amir1: done :-} [10:28:35] \o/ [10:28:37] thanks [10:28:46] Amir1: I havent been careful in rejecting that Diff, sorry :-} [10:29:35] that's totally okay thank you for accepting it [11:23:52] RECOVERY - Host integration-trusty-1026 is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [11:28:06] (03CR) 10Aashaka: "Yes, that is exactly how it seems like." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [11:30:59] (03PS1) 10Hashar: Make zuul cloner dependencies injection less verbose [integration/config] - 10https://gerrit.wikimedia.org/r/281919 [11:34:55] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [11:37:10] (03PS2) 10Hashar: Make zuul cloner dependencies injection less verbose [integration/config] - 10https://gerrit.wikimedia.org/r/281919 [11:39:19] (03CR) 10Hashar: [C: 032] Make zuul cloner dependencies injection less verbose [integration/config] - 10https://gerrit.wikimedia.org/r/281919 (owner: 10Hashar) [11:40:32] (03Merged) 10jenkins-bot: Make zuul cloner dependencies injection less verbose [integration/config] - 10https://gerrit.wikimedia.org/r/281919 (owner: 10Hashar) [12:11:54] 6Release-Engineering-Team, 6Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2183215 (10hashar) I have removed the `Ldaptestaccount123 ` user from the Gerrit `Analytics-devs` group since the password has been made public here. https:/... [12:25:12] (03PS3) 10Paladox: [MobileFrontend] Update npm to npm-node-4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/280940 [12:25:18] (03PS4) 10Paladox: [MobileFrontend] Update npm to npm-node-4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/280940 [13:24:38] (03PS2) 10Hashar: Add fabric entry to refresh slaves scripts [integration/config] - 10https://gerrit.wikimedia.org/r/273466 [13:24:53] (03PS3) 10Hashar: Add fabric entry to refresh slaves scripts [integration/config] - 10https://gerrit.wikimedia.org/r/273466 [13:25:02] (03CR) 10Hashar: [C: 032] Add fabric entry to refresh slaves scripts [integration/config] - 10https://gerrit.wikimedia.org/r/273466 (owner: 10Hashar) [13:25:44] (03PS3) 10Hashar: Test to verify skins do not run 'testextension' jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269624 (https://phabricator.wikimedia.org/T117710) (owner: 10Legoktm) [13:25:52] (03Merged) 10jenkins-bot: Add fabric entry to refresh slaves scripts [integration/config] - 10https://gerrit.wikimedia.org/r/273466 (owner: 10Hashar) [13:26:02] (03PS4) 10Hashar: Test to verify skins do not run 'testextension' jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269624 (https://phabricator.wikimedia.org/T117710) (owner: 10Legoktm) [13:26:59] (03CR) 10jenkins-bot: [V: 04-1] Test to verify skins do not run 'testextension' jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269624 (https://phabricator.wikimedia.org/T117710) (owner: 10Legoktm) [13:35:27] is deployment-puppetmaster just completely broken, hashar? [13:35:41] Krenair: how would I know ?:) [13:36:04] lol [13:39:33] PROBLEM - Puppet run on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:39:55] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:40:03] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:40:09] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:40:25] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:40:39] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:40:41] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:40:47] PROBLEM - Puppet run on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:40:55] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:41:19] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:41:20] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:41:36] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:41:38] (03PS1) 10Hashar: [MobileFronted] switch npm job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/281934 (https://phabricator.wikimedia.org/T130587) [13:41:52] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:42:20] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:42:22] PROBLEM - Puppet run on deployment-parsoid05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:42:22] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:42:36] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:42:36] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:42:44] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:42:46] (03CR) 10Paladox: [C: 031] [MobileFronted] switch npm job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/281934 (https://phabricator.wikimedia.org/T130587) (owner: 10Hashar) [13:43:28] (03CR) 10Hashar: [C: 032] [MobileFronted] switch npm job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/281934 (https://phabricator.wikimedia.org/T130587) (owner: 10Hashar) [13:43:54] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:44:00] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:44:00] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:44:00] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:44:14] (03Merged) 10jenkins-bot: [MobileFronted] switch npm job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/281934 (https://phabricator.wikimedia.org/T130587) (owner: 10Hashar) [13:44:14] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:44:14] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:46:27] Applying https://github.com/puppetlabs/puppet/commit/149b24542aa3ffaad2afef8daea051887501b684#diff-3 to deployment-puppetmaster:/usr/lib/ruby/vendor_ruby/puppet/rails/resource.rb may have fixed it [13:49:43] hashar: Hi it seems to have aborted https://integration.wikimedia.org/ci/job/mwext-mw-selenium/5388/console [13:51:30] paladox: look at the build page [13:51:38] paladox: https://integration.wikimedia.org/ci/job/mwext-mw-selenium/5388/ [13:51:55] hashar oh ok [13:51:58] thanks [13:52:27] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [13:54:08] hashar: Regarding the npm changes, im only going to do jsonlint and banana checker from now on and add jshint without making js changes. [13:55:07] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:55:27] paladox: yeah that sounds like a good idea :-} [13:55:42] paladox: I am really not a javascript guy and I might have +2 ed invalid change previously :(( [13:55:58] hashar: yep and ok. [13:56:26] hashar: I got parsoid to pass both 0.10 and 4.3 npm test [13:56:39] https://gerrit.wikimedia.org/r/#/c/280906/ [13:56:42] using if and else [13:56:59] RECOVERY - Puppet run on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:57:23] jsduck runs nativaly on trusty so we use jsduck on 0.10 whereas on npm 4.3 we will use bundle install and bundle exec jsduck. [13:57:26] hashar ^^ [13:57:27] :) [13:58:58] 5Continuous-Integration-Scaling, 3releng-201516-q4, 13Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2183399 (10hashar) [13:59:00] 5Continuous-Integration-Scaling, 10MobileFrontend, 13Patch-For-Review, 7WorkType-NewFunctionality: Migrate MobileFrontend npm job to Nodepool - https://phabricator.wikimedia.org/T130587#2183396 (10hashar) 5Open>3Resolved a:3hashar Fixed by: * adding a 'predoc' script that runs 'bundle install' * cha... [14:02:28] RECOVERY - Puppet run on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:02] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:58] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:04] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:44] RECOVERY - Puppet run on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:23] legoktm: Hi there seems to be a bug in the convert to extension.json script. Since if the php file has call_user_func init, it wont convert it. [14:09:08] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:38] RECOVERY - Puppet run on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:10:55] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:11:43] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:03] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:27] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:41] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:29] PROBLEM - App Server bits response on deployment-mediawiki03 is CRITICAL: Connection refused [14:18:54] ^ that's me [14:20:31] RECOVERY - Puppet run on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:21:00] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: Connection refused [14:22:44] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:12] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:54] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:28] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:54] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:01] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 40241 bytes in 1.248 second response time [14:34:39] Yippee, build fixed! [14:34:39] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #478: 09FIXED in 6 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/478/ [14:54:03] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [14:54:35] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:54:53] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:54:53] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [14:54:57] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:55:11] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:55:11] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:56:05] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:56:21] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:56:23] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:02:20] https://horizon.wikimedia.org/project/instances/24bfda72-2991-47bd-93d1-ce70e90171e3/ - WTF? [15:02:23] who did that? [15:04:57] (someone started the instance that was supposed to be shut down) [15:05:08] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:10] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:20] 10Beta-Cluster-Infrastructure, 6Operations, 7WorkType-NewFunctionality: etcd/confd is not started on deployment-cache-mobile04 - https://phabricator.wikimedia.org/T116224#2183569 (10Krenair) 5Open>3declined Deleting instead: {T130473} [15:06:26] PROBLEM - Host deployment-cache-mobile04 is DOWN: CRITICAL - Host Unreachable (10.68.18.110) [15:06:53] 10Beta-Cluster-Infrastructure: Finish removal of deployment-cache-mobile04 - https://phabricator.wikimedia.org/T130473#2137043 (10Krenair) Someone started the host again. I have shut it off again. [15:07:20] twentyafterfour: the poor beta-build-deb job can't build the scap .deb because it tries trie to git clone from Phabricator over ssh :-( https://phabricator.wikimedia.org/D162#4049 [15:07:32] related to your discussion with chase at https://phabricator.wikimedia.org/T131375#2178776 [15:11:25] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:12:10] Hashar: I haven't figured out why labs can't ssh to Phabricator [15:12:53] According to chase it should work but something is blocking it [15:14:50] 10Continuous-Integration-Infrastructure, 5Gerrit-Migration, 3releng-201516-q4, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2183605 (10mmodell) I'm sure we could hack the Jenkins job to use https but... [15:16:27] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:17:25] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [15:19:30] actually I guess I hadn't read all the comments there... chase provided a suggested solution [15:20:49] 10Continuous-Integration-Infrastructure, 5Gerrit-Migration, 3releng-201516-q4, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2183636 (10mmodell) Why is labs blocked from connecting to ssh? Is that to... [15:22:23] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:41] twentyafterfour: Hi have you managed to have a look at the installer i created for windows. [15:24:43] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:24:45] twentyafterfour: Could you create a repo on phabricator so i can import the code into phabricator. There needs to be two repos one for arcanist so something like arcanist/windows please since arcanist dosen't work properly on git for windows bash but upstream carn't reporduce. But i have a fix and it works. And then second repo arcanist/windows-installer please [15:25:09] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:17] paladox: aracanist is already in diffusion, so use that for that part, and file a task about the arcanist windows installer so we can look/review whats going on [15:28:08] I can't review a windows installer because I don't have windows and haven't used it since windows xp [15:28:23] but in theory it looks ok... I looked at what paladox had on github [15:29:30] greg-g: Ok, so i can upload a patch to arcanist in phabricator wikimedia that fixes it for windows users. I can switch over the submodule links from github from arcanist that i updated to the one in phabricator wikimedia. [15:29:40] twentyafterfour: Ok [15:29:55] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:56] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:45] hashar: Hi! You mentioned here https://phabricator.wikimedia.org/T130576 that you removed the account from the group because password was made public - I was wondering where it's public? [15:31:27] madhuvishy: oh yeah sorry [15:31:33] madhuvishy: seem the user/password where on the task [15:31:57] ah no it was user name and shell username i think :) [15:32:16] oh [15:32:21] must have mixed up things [15:32:26] feel free to add it back [15:33:21] okay :) [15:33:26] greg-g: and twentyafterfour: Ive created the task at https://phabricator.wikimedia.org/T131939 [15:34:47] hashar: i had to put the private key of the user on jenkins though - to load via ssh agent - and I see those are already uploaded to and accessible only via credential store [15:34:54] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:56] greg-g: and twentyafterfour: If i upload a patch to https://phabricator.wikimedia.org/diffusion/ARC/ woulden it be overwritten next time we update arcanist. [15:35:55] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 3releng-201516-q4, 7JavaScript, 13Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#2183684 (10zeljkofilipin) [15:37:52] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 3releng-201516-q4, 7JavaScript, 13Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#1532739 (10zeljkofilipin) 5Open>3Resolved @greg: I think this can be resolved. @dduvall... [15:39:12] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 3releng-201516-q4, 7JavaScript, 13Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#2183697 (10zeljkofilipin) @dduvall: I am not sure I have understood you: are you suggesting... [15:40:32] 5Gerrit-Migration, 10Phabricator: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2183701 (10Paladox) [15:43:21] paladox: no, I merge from upstream rather than just mirroring their repo [15:43:34] so far we don't have any patches on that repo but soon we probably will [15:44:05] twentyafterfour: Oh, ok, Could i upload a patch that will fix arc running on git for windows bash. Arc works in cmd but not git for windows. [15:44:07] please [15:44:13] But upstream coulden reproduce. [15:44:22] 6Release-Engineering-Team, 6Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2183715 (10madhuvishy) Spoke about this on irc already but leaving it here - Only the username of the test user was public, not the password. The commit auth... [15:44:27] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team: Postmortem: CI incidents - week of Feb 8th - https://phabricator.wikimedia.org/T126634#2183716 (10greg) 5Open>3Resolved Done enough and too old for any other details now [15:49:05] paladox: ok [15:50:10] twentyafterfour: Thanks, Ive done it here https://gerrit.wikimedia.org/r/281955 [15:50:10] would you be able to review please. [15:50:16] 5Continuous-Integration-Scaling: Write a big picture overview about what the CI isolation project is and a rough roadmap - https://phabricator.wikimedia.org/T102137#2183741 (10greg) p:5Low>3Normal Plan of action: * Antoine will write a first draft of this * Antoine and @thcipriani will pair on a reading/edit... [15:50:50] paladox: I can't really do an effective review but I will look it over [15:51:08] twentyafterfour: Ok, thanks [15:51:54] 5Continuous-Integration-Scaling: Write a big picture overview about what the CI isolation project is and a rough roadmap - https://phabricator.wikimedia.org/T102137#2183752 (10greg) https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation [16:01:10] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to puppetmaster cert - https://phabricator.wikimedia.org/T131945#2183832 (10Krenair) [16:03:24] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2183851 (10Krenair) [16:04:06] twentyafterfour: Im not sure if we can detect git for windows in arcanist. Not sure if there is code for that. [16:06:24] madhuvishy: sorry I was in a meeting and heading back home now :( [16:06:32] madhuvishy: is the ssh-agentmagic thing working ? :D [16:06:50] 5Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2183890 (10greg) [16:07:01] hashar: np! yes it is - https://github.com/wikimedia/analytics-refinery-source/commit/5f53b9f806a39dfbc2d603cde7e0323bff924e75 was a commit pushed by the testaccount123 user! [16:07:02] paladox: does git for windows set the shell environment variable to "bash" or similar? that is the most obvious way I can think of to detect it [16:07:12] madhuvishy: it is magic -:} [16:07:30] it then tried to push the tag - i'll talk to ottomata to figure out the right permissions for the user [16:07:30] madhuvishy: note how it is assigned to "chasemp" :-D [16:07:41] yes :D i stole his test account [16:07:52] madhuvishy: the release user would need to have Gerrit permission to push commit for which it is not the author/comitter [16:08:08] and we can probably have a generic account created for releasing [16:08:20] not sure if we want a generic "release" user to be used for all repos [16:08:21] twentyafterfour: Looks like it since just doing bash in git for windows runs but in cmd returns as unkown command [16:08:25] yeah - hashar right now i'm using the test account as the release user [16:08:27] or per repo specific users. Something Dan can help [16:09:00] madhuvishy: kudos really ! [16:09:04] twentyafterfour: We can also do which or where commands since which git only runs in git for windows and where only runs in cmd. [16:09:26] yup, will file a task once it all works. so i gave the test user push access - but it still can't push tags - i don't have all gerrit powers though - so will ask andrew :) [16:09:28] thanks! [16:11:28] twentyafterfour: Would you know the code we could use to do that please [16:11:43] madhuvishy: or releng folks can assist with Gerrit permissions. We are most probably all Gerrit admins [16:13:45] not me! [16:13:48] :P [16:14:38] paladox: I'm not sure. inside git for windows, what do you see when you run `echo $shell` [16:15:08] twentyafterfour: Nothing just blank [16:15:15] $ echo $shell [16:15:26] or echo $SHELL [16:15:39] (I guess it needs to be capitalized) [16:15:49] twentyafterfour: It shows [16:15:50] $ echo $SHELL [16:16:13] twentyafterfour: It works is what i meant to say [16:16:23] twentyafterfour it shows [16:16:24] $ echo $SHELL [16:16:47] usr/bin/bash [16:17:06] twentyafterfour: Sorry about that any thing that has / dosen't work [16:17:14] if it is at the front on hexchat [16:17:17] hashar: I'm trying to reserve getting releng folks to do things for harder things :D [16:17:30] $ echo $SHELL [16:17:30] /usr/bin/bash [16:19:42] 10Continuous-Integration-Config, 10MediaWiki-extensions-DonationInterface: jjb: run composer install in DonationInterface - https://phabricator.wikimedia.org/T131264#2183905 (10JanZerebecki) If you do not need the features composer.lock provides then using the merge plugin (which will ignore the lock ) will cr... [16:23:03] 10Browser-Tests-Infrastructure, 13Patch-For-Review: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2183909 (10zeljkofilipin) [16:24:05] Dereckson: Hi how would i do uname in php. So i can do an if and else statement so if uname dosen't work it goes to else. [16:24:07] please [16:24:20] And thanks for researching it. [16:26:18] https://secure.php.net/manual/en/function.posix-uname.php [16:26:48] paladox: https://github.com/wikimedia/mediawiki/blob/master/includes/GlobalFunctions.php#L1486-L1515 [16:27:04] Reedy: Thanks [16:28:20] thcipriani: , twentyafterfour, ithink i'dl like to try and merge those patches today [16:31:43] Yippee, build fixed! [16:31:44] Project selenium-PdfHandler » firefox,test,Linux,,contintLabsSlave && UbuntuTrusty build #16: 09FIXED in 42 sec: https://integration.wikimedia.org/ci/job/selenium-PdfHandler/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,VERSION=,label=contintLabsSlave%20&&%20UbuntuTrusty/16/ [16:36:57] Reedy, paladox > it's probably a bit tricky here, as PHP is compiled directly for Windows (so PHP_OS is WINNT) but the real question is "It is launched from cmd.exe or from MinGW/MSYS/Cygwin?" [16:37:16] (or from a shell under Mi...) [16:38:19] Dereckson i doint think in bash on windows it loads from cmd. sinc using which in git for windows bash works but in cmd it dosen't and which is a linux command not a windows. Whereas where is a windows command [16:38:20] Arcanist has a fix "for Windows" to correct a queer behavior about pipelines [16:38:27] but this is a behavior in cmd.exe [16:40:56] Dereckson yep [16:42:25] In https://secure.phabricator.com/T3266 the Bash installed by Git is used (a bash prompt + "Msysgit" in the task description). [16:42:43] Strange, that's the same config you're using. [16:42:44] ottomata: cool with me, let me know how I can help [16:43:03] Dereckson would doing [16:43:04] if ( shell_exec('which git') ) { [16:43:11] work since it works for me [16:43:24] if (phutil_is_windows()) { [16:43:24] if ( shell_exec('which git') ) { [16:43:24] $git = 'git'; [16:43:24] } else { [16:43:24] // NOTE: On Windows, phutil_passthru() uses 'bypass_shell' because [16:43:25] // everything goes to hell if we don't. We must provide an absolute [16:43:27] // path to Git for this to work properly. [16:43:31] $git = Filesystem::resolveBinary('git'); [16:43:33] $git = csprintf('%s', $git); [16:43:35] } [16:43:37] } else { [16:43:39] $git = 'git'; [16:43:41] } [16:43:43] } [16:44:16] Oh wait that breaks it for cmd. [16:44:21] https://phabricator.wikimedia.org/paste/ [16:44:31] There is `where` for Windows / cmd.exe. [16:45:02] or `arc paste` [16:45:17] even easier [16:46:54] 5Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2184013 (10greg) Ugh, http://www.jrsoftware.org/files/is/license.txt is a weird combo of [[ https://opensource.org/licenses/BSD-3-Clause | 3 clause BSD ]] and [[ https://opensource.or... [16:47:03] marxarelli: :) [16:48:22] 5Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2184021 (10Paladox) @greg yep, but it shoulden cause license issues for us since we are not including the whole programme just using it to generate the file that it can then create th... [16:50:00] Speaking of `arc paste`, I like https://phabricator.wikimedia.org/P2868 and https://phabricator.wikimedia.org/P2869 scripts. [16:51:04] I wrote the first to avoid to always write `cat file.txt | arc paste --title file.txt`, the second to diff two different pastes. [16:51:57] 5Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2184033 (10greg) Yeah, mostly just an aside from a license geek :) Nothing in there restricts what you can do with generated binaries, afaict. [16:57:27] (03CR) 10Zfilipin: [C: 04-1] "VisualEditor can not find ci.yml :(" (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/275820 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [17:00:46] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 3releng-201516-q4, 7JavaScript, 13Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#2184041 (10dduvall) >>! In T108874#2183697, @zeljkofilipin wrote: > @dduvall: I am not sure... [17:04:37] twentyafterfour and Dereckson unbuntu has been added to windows as of today preview users can get it [17:04:37] https://blogs.windows.com/windowsexperience/2016/04/06/announcing-windows-10-insider-preview-build-14316/ [17:04:39] :) [17:04:48] Im going to download it [17:05:14] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 3releng-201516-q4, 7JavaScript, 13Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#2184044 (10greg) Well reasoned/said, Dan. [17:06:04] paladox: just to be sure: we don't want to make the de fact way of installing arcanist on windows require Win 10+Ubuntu, just fyi [17:06:33] greg-g: yep, i will still use git for windows. [17:07:01] * greg-g nods [17:08:28] 6Release-Engineering-Team, 15User-greg: Create 1516Q3 quarterly review slide deck - https://phabricator.wikimedia.org/T131954#2184082 (10greg) [17:18:40] 5Gerrit-Migration, 10Differential: Cross-repository gating of changes pre-merge in Differential - https://phabricator.wikimedia.org/T131955#2184133 (10greg) [17:24:27] marxarelli: can I add a plugin to Jenkins from the UI? [17:24:36] 5Gerrit-Migration, 10Differential: Cross-repository gating of changes pre-merge in Differential - https://phabricator.wikimedia.org/T131955#2184177 (10greg) [17:25:18] this one - https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin - It's on the available list - but not installed [17:25:44] Dereckson and Reedy doing https://phabricator.wikimedia.org/P2871 works but in git for windows it shows the where command running so it shows that the command is undefined in git but the command works on cmd. [17:25:56] How can i run it without showing or is there something similar [17:26:46] twentyafterfour ^^ [17:28:21] paladox: that should work, what do you mean by 'without showing' ? [17:28:44] twentyafterfour: [17:28:44] $ arc land [17:28:44] Landing current branch 'master'. [17:28:44] TARGET Landing onto "master", selected by following tracking branches upstream to the closest remote. [17:28:44] REMOTE Using remote "origin", selected by following tracking branches upstream to the closest remote. [17:28:47] FETCH Fetching origin/master... [17:28:49] 'where' is not recognized as an internal or external command, [17:28:51] operable program or batch file. [17:28:53] Usage Exception: There are no commits on "master" which are not already present on the target. [17:28:55] Sorry i should have pasted that in paste [17:29:15] But it does 'where' is not recognized as an internal or external command, [17:30:16] 10Continuous-Integration-Infrastructure, 5Gerrit-Migration, 3releng-201516-q4, 10Differential: Spec out needed glue for Differential to Gearman to Nodepool - https://phabricator.wikimedia.org/T130949#2184185 (10greg) Pasting here so it's not lost... An email from Antoine about Gearman examples in PHP: >... [17:33:36] 6Release-Engineering-Team, 15User-greg: Create person/project heat map - https://phabricator.wikimedia.org/T131764#2184190 (10greg) https://www.mediawiki.org/wiki/User:Greg_(WMF)/Sandbox [17:34:07] paladox: so the SHELL environment variable wasn't available? what you are trying to do with 'where' doesn't seem reliable [17:34:32] twentyafterfour: It shows that on git for windows but works on cmd. [17:34:35] I'd try getenv('SHELL') to see if it's set to 'some\path\to\git-bash.exe' [17:34:44] twentyafterfour: Ok [17:34:53] Thanks i will try now [17:35:12] that way other shells on windows won't break [17:35:37] also the best way forward is probably the new ubuntu on windows thing you are trying out ;) [17:36:42] madhuvishy: i believe we only install plugins from debs, not completely sure [17:36:53] marxarelli: aah [17:37:10] twentyafterfour: Yeh, seems to be a big update to windows has ton of new features [17:37:43] paladox: updated your diff: https://phabricator.wikimedia.org/P2871 [17:37:55] maybe that will work? [17:38:58] twentyafterfour: Thanks [17:39:49] marxarelli: who to ask :) hashar? [17:41:43] madhuvishy: hashar is your best bet or search phab for closed tasks about installing other plugins [17:42:01] marxarelli: okay cool will do that [17:42:13] madhuvishy: and subscribe me please! i would like to know :) [17:42:32] madhuvishy: file a task and subscribe me that is [17:42:41] yup! will do :) [17:42:50] rad, thanks! [17:49:52] 10Continuous-Integration-Infrastructure: Jenkins: Install Config File Provider Plugin - https://phabricator.wikimedia.org/T131958#2184241 (10madhuvishy) [18:20:39] 10Continuous-Integration-Infrastructure, 5Gerrit-Migration, 3releng-201516-q4, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2165259 (10Andrew) > Why is labs intentionally blocked from connecting to s... [19:06:16] twentyafterfour: Sorry it took this long to get back to you but ive updated the diff at https://phabricator.wikimedia.org/P2871 that change dosent work since it would use the same as before except from an extra check but https://phabricator.wikimedia.org/P2871 works for me [19:06:22] See the new diff section please [19:20:10] twentyafterfour: Ive updated the commit at https://gerrit.wikimedia.org/r/#/c/281955/ [19:20:15] Dereckson ^^ [19:22:53] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:05] twentyafterfour: am i getting ready to merge stuff [19:24:12] you aroudn to verify that things are ok? [19:39:59] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184615 (10dduvall) [19:40:20] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184599 (10dduvall) [19:40:22] 6Release-Engineering-Team, 5Release: MW-1.27.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T130945#2184616 (10dduvall) [19:41:07] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184599 (10Reedy) https://gerrit.wikimedia.org/r/281986 [19:42:30] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2183851 (10Andrew) I tidied up apt a bit on that instance (which shouldn't have been r... [19:42:53] Can I get someone who cares about deployment-prep and puppet to look at ^ ? [19:43:25] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2184625 (10Andrew) Of course the more upsetting error that Krenair saw may resurface a... [19:45:23] andrewbogott: I can take a look [19:45:44] thcipriani: it might be as simple as unchecking a box, but… there's probably more to it than that [19:45:53] there always is. [19:47:06] 5Continuous-Integration-Scaling, 6Operations, 7HHVM: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia - https://phabricator.wikimedia.org/T125821#1997890 (10akosiaris) So, up to now we did not have to package HHVM for jessie-wikimedia. I don't have an ETA on when it will be re... [19:56:09] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [19:59:03] hashar: I made this task - https://phabricator.wikimedia.org/T131958 - could you take a look? [20:04:27] yeehaw, thcipriani, twentyafterfour. applied those patches on tin and mira! [20:04:38] i had to remove the phabricator-deploy stuff, something wasn't working with that [20:04:44] twentyafterfour: if you come back around, let's figure that out [20:05:00] ottomata: nice! Saw that you got off to a bit of a rocky start :) [20:05:18] glad you were able to get it worked out! That's awesome. [20:07:03] madhuvishy: not really around tonight beside random chit-chat and reading the news. But I have poked our team channel :-} [20:07:57] hashar: okay :) thanks! and have a nice evening [20:09:51] madhuvishy: actually you should be to add it -:} [20:10:01] hashar: just from the UI? [20:10:13] aren't you a WMF employee? ;-) [20:10:15] i wasn't sure if that's okay to do [20:10:18] i am! [20:11:06] 6Release-Engineering-Team, 5Release: MW-1.27.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T130945#2184681 (10greg) [20:11:08] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184679 (10greg) 5Open>3Resolved a:3Reedy [20:11:18] 10Continuous-Integration-Infrastructure: Jenkins: Install Config File Provider Plugin - https://phabricator.wikimedia.org/T131958#2184682 (10hashar) We used to have a maven configuration file on the CI slaves to point maven to use a web proxy. That got removed though. Puppet is a bit annoying anyway and one has... [20:11:28] madhuvishy: ^^^ [20:11:41] tasks are cheap :-}  That is usually the best way to ask around (beside IRC of course) [20:11:42] hashar: coool [20:11:45] so I guess yeah: go for it ! [20:11:52] DO NOT UPGRADE OTHER PLUGINS !!!!!!! ;D [20:11:58] I will not :D [20:12:28] * Reedy upgrades everything [20:12:33] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2183851 (10thcipriani) The `role::cache::parsoid` error likely stems from rOPUP6d215ed... [20:12:54] madhuvishy: that is a good finding really [20:13:21] hashar: ya definitely looks useful. It got installed too - now to test :) [20:14:11] yeah and sometime plugins do not even require a restart [20:14:50] 10Continuous-Integration-Infrastructure: Jenkins: Install Config File Provider Plugin - https://phabricator.wikimedia.org/T131958#2184690 (10madhuvishy) p:5Triage>3Normal a:3madhuvishy [20:14:54] 10Continuous-Integration-Infrastructure: Jenkins: Install Config File Provider Plugin - https://phabricator.wikimedia.org/T131958#2184230 (10madhuvishy) 5Open>3Resolved Thanks @hashar - Installed it :) [20:18:20] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2184700 (10Krenair) It's more like an ongoing battle than a task we can perform once a... [20:18:43] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:18:57] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:19:07] madhuvishy: grats! thanks for the pingç [20:19:17] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:19:53] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:21:03] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:23:57] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:24:12] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:39:05] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:19] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:41:03] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:41:42] 10Continuous-Integration-Infrastructure, 5Gerrit-Migration, 3releng-201516-q4, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2184719 (10chasemp) 22 to only 208.80.154.250/32 as the service address for... [20:47:45] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #977: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/977/ [20:49:06] RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:52] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184734 (10Florian) Sorry :( [20:51:04] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184599 (10Krenair) Please be careful not to break extensions being deployed from non-master branches when performing such changes [20:51:28] Krenair: Shit happens, unfortunately [20:51:32] It's pretty much the only exception [20:51:44] SMW is the only extension not being deployed from master? [20:52:21] Wikidata, but that's special [20:52:25] SRF and Validator too [20:52:28] But they're related to sMW [20:52:39] And then CN uses their deploy branch [20:54:53] So basically you can completely break all wikis by forgetting about the special extensions [20:55:38] well, 60% of them are Wikitech only [20:55:44] Wikidata isn't going to break [20:55:50] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184599 (10hashar) That is a bit hard to catch though. One would have to test / run tests with SemanticMediaWiki `1.8` branch. Maybe we can craft a Jenkins job for that? [20:55:55] Meaning, it's possibly just CN [20:56:16] potentially we could have a Jenkins job that get mediawiki@master + SemanticMediaWiki @ 1.8 + whatever dependencies are needed [20:56:18] https://github.com/wikimedia/mediawiki-extensions-CentralNotice/tree/wmf_deploy [20:56:20] and run the SMW tests [20:56:22] "This branch is 117 commits ahead, 44 commits behind master." [20:56:35] eek [20:56:59] I wouldn't worry about CN too much :) [20:57:55] and SMW will probably be removed if OpenStackManager is removed / no more used on wikitech [20:58:05] yup [20:58:20] at least a breakage has minimal impact, it would solely be wikitech [20:58:24] and/or active functionality is slimmed down [21:04:13] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2183851 (10greg) "Things broken by puppet changes not tested in Beta Cluster"? :) [21:14:09] 6Release-Engineering-Team, 5Release: Call to undefined method Title::newFromRedirect - https://phabricator.wikimedia.org/T131973#2184838 (10hashar) /me conforts @Florian . The impact is minimal anyway here, it is "solely" wikitech so that is only for a few people who most probably knows how to quickly escalat... [21:25:34] some general question to the CI freaks ;). Why we have non-voting tests, what's the advantage of them ? [21:26:55] FlorianSW: Hi did you work on add support for composer in extension registration. [21:27:26] paladox: what is "support for composer in extension registration"? o.O [21:27:33] Luke081515: The advantage is someone one day will fix the test so we can make them voting [21:27:50] FlorianSW: https://phabricator.wikimedia.org/T119766 [21:28:03] paladox: So a non voting test ist just a test, which isn't fully written correctly? [21:28:42] Luke081515 yes its a test just for repos that are currently failing and no one is going to fix them near them. [21:29:00] ok, thx [21:29:40] your welcome [21:29:42] paladox: that's not entirely correct [21:30:33] a non-voting test is used whenever you don't want to give a -1 when the test is failing. One use case could be, that the repo currently fails a (new) test suite, but doesn't mean, that the extension will never pass the test. [21:30:53] FlorianSW Oh ok [21:31:01] Luke081515 ^^ [21:33:00] paladox: and to the composer thing: I usually don't work on things that aren't broken. Please, if you've a problem with a feature, open a new task, and don't reopen old ones :) [21:33:15] FlorianSW: Ok [21:35:20] 10Continuous-Integration-Infrastructure, 6Multimedia, 6Operations, 13Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#2184918 (10hashar) OpenStack enquired about imagemagick on Trusty requiring ffmpeg. But ffmpeg go... [21:36:03] ok, thx [21:37:54] FlorianSW: https://phabricator.wikimedia.org/T131978 [21:42:59] 5Continuous-Integration-Scaling, 6Operations, 7HHVM: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia - https://phabricator.wikimedia.org/T125821#1997890 (10greg) One thought from Alex in the SoS was creating a trusty nodepool image for these tests (composer) to unblock us (Re... [21:44:51] paladox: I'm not sure, but I don't see anything that looks related to the composer autoloading. What about the errors at the top of the the build: https://travis-ci.org/JeroenDeDauw/Maps/jobs/121273122#L806-L868 ???? [21:45:46] paladox: in the extension.json I don't see any AutoloadClasses, is that right? I don't think so. [21:45:58] FlorianSW: Thats because it didn't find the autoload.php file in vendor folder with those classes added in composer. [21:46:02] FlorianSW: Yes [21:46:18] FlorianSW thats because they are loaded through composer [21:47:56] pff, you can't mix these things... [21:48:42] FlorianSW: But it works and yes you can. Composer supports adding classes. [21:49:00] There are some extensions that doint use autoloading like wikibase and wikidata [21:49:23] 5Gerrit-Migration, 5Gitblit-Deprecate, 6Release-Engineering-Team, 10releng-201516-q3, and 4 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#2184969 (10greg) >>! In T119908#2176799, @JanZerebecki wrote: > T130949 implies that zuul will b... [21:51:47] paladox: no you can't ;) At least not, if you rely on the callback in extension registration, as the callback is executed before the extra autoload paths (like the composer autoloader) are loaded. [21:52:24] FlorianSW: Oh, but Wikibase and wikidata do that. [21:52:31] Autoload through composer [21:52:58] I'm pretty sure wikibase doesn't use extension registration, right? [21:53:33] paladox: ^ [21:53:36] 5Continuous-Integration-Scaling, 6Operations, 7HHVM: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia - https://phabricator.wikimedia.org/T125821#2185010 (10hashar) Potentially we could generate an image based on Trusty then I would rather switch all of CI to run solely on Deb... [21:53:43] FlorianSW yes [21:53:56] so ;) [21:54:14] that's what I'm talking about. Currently, Wikibase wouldn't work with extension registration, too. [21:57:28] FlorianSW: Oh ok [21:58:13] FlorianSW: But then shoulden we make composer vendor/autoload.php load earler [22:02:19] Dereckson: Hi would getenv('SHELL') === '/usr/bin/bash [22:02:19] ' work [22:02:27] twentyafterfour ^^ [22:02:47] Thats for supporting other clients that works on windows but base on linux [22:16:25] 6Release-Engineering-Team, 6Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2185058 (10madhuvishy) More things! 11. It failed saying it couldn't push a tag because the test user needed Push Annotated Tag permissions in Gerrit. Added... [22:19:10] marxarelli: i was celebrating on #analytics a bit back - I got jenkins to release the jars :D [22:20:20] Is this where we complain about zuul? I'm gonna go ahead and force submit all my patches, unless there are objections... [22:21:46] awight: one moment [22:22:10] awight: please don't, things are almost done [22:22:12] https://integration.wikimedia.org/zuul/ [22:22:21] blame wikibase tests taking too long :) [22:22:26] Dereckson ive updated the commit at https://gerrit.wikimedia.org/r/#/c/281955/ [22:22:32] twentyafterfour ^^ [22:22:34] works now [22:22:52] paladox: you don't need to ping people in IRC after you update a patch, they get notified in Gerrit [22:22:53] gah so sorry, I just did [22:23:01] greg-g: Ok sorry [22:23:07] greg-g: ^ Got your msg too late :( [22:23:19] awight: seriuosly, don't come in here and do that [22:23:27] I responded in 1 minute [22:23:47] gah, gate-and-submit is pretty full [22:24:23] Luke081515: it'll be fine, just let that wikibase one finish [22:24:28] zuul is going to crash now unless that only happends when including dependacys from repos not registered in zuul [22:24:36] yeah, that test has currently 61% [22:24:39] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [22:24:40] https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/1484/console [22:26:10] Luke081515 scribuntu is slowing it down. [22:26:22] :-/ [22:26:26] greg-g: Yeah that was terrible. I hardly have an excuse, just got antsy about waiting for the wikibase tests, with no guarantee that I would get a zuul-submit even when they were done. [22:26:42] * greg-g nods [22:27:48] so this would be a better place to ask--why is there a non-parallel step in the gate-and-submit queue? [22:28:07] cross repo dependencies [22:28:08] The repos are completely independent, it's really strange that we would have to wait for other people's tests [22:28:11] gah [22:28:23] ish, it's probably an indirect dependency [22:28:27] just a general assumption of dependencies, or are they specifically mapped? [22:28:33] specifically [22:28:43] we don't have fancy AI stuff to do it for us like Google :) [22:30:23] related: https://phabricator.wikimedia.org/T126670 [22:30:29] the wikibase tests are a wildly variable duration... https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/buildTimeTrend [22:30:46] I wonder if there's some blocking stuff in there [22:31:14] greg-g: I think adding dependacys from other repos that have not be registered to zuul yet have been fixed upstream. [22:31:28] So once zuul is upgraded those types of errors should not happen [22:31:29] awight: that's not just wikibase :) [22:31:31] * awight wanders back to making other people's lives miserable ;) [22:31:42] awight: godspeed [22:32:38] I hate that I can't take back the spherically-shaped firework in the plumbing [22:33:15] awight: Only 3% to go at that slow test ;) [22:33:46] paladox: I would appreciate if you wouldn't amend my changes, especially if I'm online and see what jenkins says to me ;) [22:33:50] ... and then all my force submits will run into a mysterious brick wall somewhere in zuul :( [22:34:04] FlorianSW Ok sorry. It wont happend any more [22:34:37] paladox: generally it's best to left the change author do that [22:34:46] greg-g: Ok [22:35:50] hm, parsoid is using a seperate queue [22:36:14] Luke081515 yes because it creates its own custom tests [22:36:25] So its out of the queue [22:36:30] hm, ok [22:36:57] madhuvishy: awesome! nice work [22:37:13] Luke081515 if you want you can add the queue to parsoid [22:37:18] 6Release-Engineering-Team, 6Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2139858 (10greg) well done, @madhuvishy! (and @dduvall!) [22:37:36] marxarelli: thank you :) and for all the help! I'll make more tasks to bother y'all soon. [22:37:47] madhuvishy: :) :) [22:37:51] for sure. that's what we're here for [22:37:56] * Luke081515 is no test/CI expert [22:38:01] :D [22:38:26] Luke081515: I can do it. Would you like it done. [22:39:13] better don't listen to me, I have no idea :P [22:39:21] I'm better in phabricator things [22:40:31] Luke081515: Oh ok [22:46:33] bd808: sorry I wasn't able to do more other than "shut 'er down" when stashbot was misbehaving on Monday [22:47:22] greg-g: no worries. The Tool labs elastic cluster was busted. [22:47:31] I got it sorted yesterday afternoon [22:48:49] like most bots, it's not too resilient in the face of infrastructure failures yet [22:49:37] which is our only hope in the face of annihilation [22:50:04] ok, things look to be progressing safely in zuul-land, thankfully [22:50:31] "Today humanity was saved by a flapping DNS route" [22:52:08] "Our plan for tomorrow is similarly a long shot." [22:54:19] twentyafterfour: arc would work on native bash on windows because even though its on windows its live translating linux to windows and is sandboxed. [22:55:15] Im not sure if it is sandboxed now. But i can use ruby and python made for linux now. [22:56:28] define sandboxed? my understanding is that you have access to the entire system (modulo your user rights) from the Ubuntu bash (and vice-versa) [22:57:29] Is all this windows madness just because the script uses a #!/usr/bin/php or something? [22:57:55] can't there just be an arc.cmd script added upstream? [22:58:08] greg-g: Well it is limited i think [22:58:45] greg-g: I read here http://www.winbeta.org/news/native-bash-ubuntu-comes-windows-10-insiders-build-14316 that you can use your other tools in it too plus access your windows files. [22:59:16] yes [22:59:34] bd808 no and there cmd.bat but git for windows in bash dosen't work but i fixed it with https://gerrit.wikimedia.org/r/#/c/281955/ [22:59:52] bd808 i reported upstream but they said they could not reproduce and closed the task. [23:02:21] Im not sure but it looks like cpanel can now be installed on windows without them supporting it which they droped years ago