[00:02:00] (03PS2) 10Jforrester: WIP: Test MW-core against PHP55 for master/REL1_27+ [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) [00:02:11] !log integration-slave-trusty-1016:~$ sudo -i rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/skins/Donate [00:02:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:02:49] (03CR) 10jenkins-bot: [V: 04-1] WIP: Test MW-core against PHP55 for master/REL1_27+ [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:03:08] 10Continuous-Integration-Infrastructure: some tests run from mwext-testextension-hhvm will pick up files from extensions that were not checked out for this job - https://phabricator.wikimedia.org/T117710#1972150 (10JanZerebecki) `integration-slave-trusty-1016:~$ sudo -i rm -rf /mnt/jenkins-workspace/workspace/mw... [00:04:51] (03PS1) 10Dduvall: Target testwiki for PdfHandler browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/266952 [00:10:43] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1972188 (10Andrew) 3NEW a:3hashar [00:12:58] (03CR) 10JanZerebecki: WIP: Test MW-core against PHP55 for master/REL1_27+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:13:41] (03CR) 10Jforrester: WIP: Test MW-core against PHP55 for master/REL1_27+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:13:55] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1972220 (10Reedy) https://gerrit.wikimedia.org/r/#/c/266942/ should be merged, but it shows a bigger problem. A dependency on LdapAuthentic... [00:14:13] legoktm: Don't stress yourself out, but… https://gerrit.wikimedia.org/r/#/c/266941/ :-) [00:16:36] (03CR) 10JanZerebecki: WIP: Test MW-core against PHP55 for master/REL1_27+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:16:42] heh [00:17:25] OK… [00:17:40] legoktm: Want to take it over or should I keep fiddling? ;-) [00:17:51] Get jzerebecki to do it :D [00:18:12] Ha. [00:19:58] 10Beta-Cluster-Infrastructure, 6Services: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972271 (10mobrovac) [00:20:29] James_F: so subtle :) [00:20:50] hey people, is VE not enabled in beta? [00:21:17] it is for me... [00:21:26] (03PS3) 10Jforrester: WIP: Test MW-core against PHP55 for master/REL1_27+ [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) [00:21:42] mobrovac: Which wiki? [00:21:48] http://en.wikipedia.beta.wmflabs.org/wiki/Lucius_Arruntius [00:22:12] works for me. [00:22:20] (03CR) 10jenkins-bot: [V: 04-1] WIP: Test MW-core against PHP55 for master/REL1_27+ [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:22:23] mobrovac: It mirrors production config, so it's only available to logged-in accounts that haven't opted out. [00:22:53] James_F: it didn't appear in my prefs, but adding ?veaction=edit fixed it [00:22:54] thnx [00:23:05] mobrovac: http://he.wikipedia.beta.wmflabs.org/wiki/%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:Jdforrester_(WMF)/Sandbox has VE even for IPs, for instance. [00:23:06] Sure. [00:23:44] mobrovac: you should have the following in prefs, editing section [00:23:54] Temporarily disable the visual editor while it is in beta [00:24:10] Editing tabs: dropdown with options [00:24:42] * mobrovac checking [00:26:18] all good now, the dropdown confused me :P [00:26:20] thnx thedj [00:26:54] "the dropdown confused me" then that might be something that should be evaluated [00:27:48] to be exact, "editing tabs" didn't feel as something important that would allow me enable VE [00:27:57] s/me/me to/ [00:31:57] 10Beta-Cluster-Infrastructure, 6Services, 6operations: Move Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T124989#1972352 (10mobrovac) [00:32:36] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1971975 (10mobrovac) [00:35:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1972364 (10scfc) [00:35:19] 10Continuous-Integration-Config, 10MediaWiki-extensions-OpenStackManager: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#1972365 (10scfc) [00:35:52] mobrovac: It's enabled already, but understood. [00:36:45] !log deployment-prep re-imaging deployment-mathoid for T125003 [00:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:36:56] *sigh* github is down [00:37:28] forced holiday! [00:37:28] jzerebecki: This is yet another reason why we don't depend on them. [00:39:13] James_F: it seems many people working for WM don't agree... [00:39:26] 6Release-Engineering-Team, 6operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#1972401 (10Dzahn) [00:39:50] github is useful for the search [00:40:40] PROBLEM - Host deployment-mathoid is DOWN: CRITICAL - Host Unreachable (10.68.17.222) [00:41:50] 6Release-Engineering-Team, 6operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#1972417 (10Dzahn) [00:43:14] 6Release-Engineering-Team, 6operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#1972435 (10Reedy) ytterbium is nearly out of warranty. I suspect as it's a still a working machine, it would go into spare after a reinstall? [00:43:18] jzerebecki: got any experience with the jenkins slaves for cxserver and parsoid perhaps? [00:44:17] mobrovac: nope, why are you asking? [00:45:08] jzerebecki: they are jenkins slaves that get updated on commit merges, and i need to reimage their boxes, cf https://phabricator.wikimedia.org/T125003 [00:47:17] mobrovac: yes. what is you question? [00:47:34] s/you/your/ [00:48:06] jzerebecki: the question is: can i just delete those machines and create new ones with the same without consequences for the automatic jenkins jobs [00:48:07] ? [00:56:05] mobrovac: the jobs are just bound to a label. so yes. but perhaps just disable the old ones in jenkins until the new ones work instead of deleting the instances beforehand. some docs here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup perhaps extend it. [00:58:21] mobrovac: i.e. you can use a new name (count up) and apply the same label when adding the instance to jenkins. [00:58:44] hmmm [00:58:45] ok [00:58:46] (reusing the name confuses ssh and other things) [00:58:47] thnx jzerebecki [00:59:11] jzerebecki: "new name" you mean for the MV names or job names? :P [00:59:31] mobrovac: host name [00:59:45] kk [01:00:48] 6Release-Engineering-Team, 6operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#1972551 (10Dzahn) @Robh please see Reedy's question above. [01:01:57] 7Blocked-on-RelEng, 6operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1972554 (10Dzahn) [01:03:23] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972559 (10JanZerebecki) 01:48 < mobrovac> jzerebecki: the question is: can i just delete those machines... [01:05:41] RECOVERY - Host deployment-mathoid is UP: PING OK - Packet loss = 0%, RTA = 1.05 ms [01:10:34] 6Release-Engineering-Team, 6operations: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#1972595 (10RobH) This would invalidate once T123132 is approved, and I'd instead reclaim ytterbium to spares. [01:12:42] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972597 (10mobrovac) [01:22:36] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #500: 04STILL FAILING in 35 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/500/ [01:27:34] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972625 (10mobrovac) [01:28:12] !log deployment-prep recreating deployment-sca01 for T125003 [01:28:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:29:02] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce build #284: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/284/ [01:31:20] PROBLEM - Host deployment-sca01 is DOWN: CRITICAL - Host Unreachable (10.68.17.54) [01:32:15] all good shinken-wm, don't worry [01:42:40] !log deployment-prep recreating deployment-sca02 for T125003 [01:42:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:48:16] PROBLEM - Host deployment-sca02 is DOWN: CRITICAL - Host Unreachable (10.68.16.173) [01:58:32] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972653 (10mobrovac) [01:59:27] RECOVERY - Host deployment-sca02 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [02:20:34] 7Browser-Tests, 5Patch-For-Review: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1972729 (10Riovilogdacusin) [02:23:51] !log deployment-prep deployment-restbase02 disabled puppet to recreate deployment-restbase01 for T125003 [02:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:27:52] !log deployment-prep recreating deployment-restbase01 for T125003 [02:27:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:47:52] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:47:54] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:49:25] !log deployment-prep deployment-restbase01 disabled puppet to set up cassandra for [02:49:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:57:51] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:10:05] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 5 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972843 (10mobrovac) [03:18:25] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Services: deployment-restbase01 is on sid; puppet-run failing as a result - https://phabricator.wikimedia.org/T113965#1972845 (10mobrovac) 5Open>3Resolved a:3mobrovac I have recreated `deployment-restbase01` from scratch, it's on Jessie now. [03:25:53] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Services: iptables rules on deployment-restbase02 are wrong - https://phabricator.wikimedia.org/T125038#1972853 (10mobrovac) 3NEW [03:41:34] 10Beta-Cluster-Infrastructure, 6Labs, 6operations: Duplicate IP address DNS entry - https://phabricator.wikimedia.org/T125040#1972870 (10mobrovac) 3NEW [03:47:50] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Services: iptables rules on deployment-restbase02 are wrong - https://phabricator.wikimedia.org/T125038#1972890 (10mobrovac) Note that the rules have been set up correctly on `deployment-restbase01`. [03:48:00] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 10RESTBase, 6Services: iptables rules on deployment-restbase02 are wrong - https://phabricator.wikimedia.org/T125038#1972892 (10mobrovac) [03:49:55] !log deployment-prep re-enabled puppet on deployment-restbase0x [03:50:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:57:21] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 5 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1972968 (10mobrovac) [03:57:35] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1971975 (10mobrovac) [03:58:20] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 4 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#1971975 (10mobrovac) All done except the jenkins slaves (Parsoid and CXServer). @hashar, I'd love your he... [05:55:46] *sigh* [05:55:54] !log doing https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update [05:55:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [06:22:18] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-12-01: [Task] Add Wikidata to extension-gate in CI - https://phabricator.wikimedia.org/T96264#1973158 (10JanZerebecki) [06:25:28] (03CR) 10JanZerebecki: "Now passes." [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) (owner: 10JanZerebecki) [06:25:39] (03CR) 10JanZerebecki: "See https://gerrit.wikimedia.org/r/#/c/266990/ ." [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) (owner: 10JanZerebecki) [06:25:47] (03PS2) 10JanZerebecki: add Wikidata to extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) [06:29:35] (03CR) 10JanZerebecki: [C: 032] add Wikidata to extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) (owner: 10JanZerebecki) [06:31:26] (03Merged) 10jenkins-bot: add Wikidata to extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) (owner: 10JanZerebecki) [06:36:07] hi [06:36:34] i am new to wiki and want to contribute towards wikimedia as my gsoc project [06:36:41] where can i start? [06:41:45] (03PS3) 10JanZerebecki: Remove dependancy on DataValues as it's been removed from Gerrit... [integration/config] - 10https://gerrit.wikimedia.org/r/265179 (https://phabricator.wikimedia.org/T93735) (owner: 10Reedy) [06:42:10] i am new to wiki and want to contribute towards wikimedia as my gsoc project [06:42:17] where can i start? [06:49:34] (03CR) 10JanZerebecki: "Deployed to Jenkins: (['mediawiki-extensions-hhvm', 'mediawiki-extensions-php53', 'mediawiki-extensions-php55', 'mediawiki-extensions-quni" [integration/config] - 10https://gerrit.wikimedia.org/r/251138 (https://phabricator.wikimedia.org/T96264) (owner: 10JanZerebecki) [06:49:51] (03PS4) 10JanZerebecki: Remove dependancy on DataValues as it's been removed from Gerrit... [integration/config] - 10https://gerrit.wikimedia.org/r/265179 (https://phabricator.wikimedia.org/T93735) (owner: 10Reedy) [06:51:52] (03CR) 10JanZerebecki: "Deployed to Jenkins: (['mwext-Validator-testextension-hhvm', 'mwext-Validator-testextension-php53'])" [integration/config] - 10https://gerrit.wikimedia.org/r/265179 (https://phabricator.wikimedia.org/T93735) (owner: 10Reedy) [06:51:58] (03CR) 10JanZerebecki: [C: 032] Remove dependancy on DataValues as it's been removed from Gerrit... [integration/config] - 10https://gerrit.wikimedia.org/r/265179 (https://phabricator.wikimedia.org/T93735) (owner: 10Reedy) [06:54:51] (03Merged) 10jenkins-bot: Remove dependancy on DataValues as it's been removed from Gerrit... [integration/config] - 10https://gerrit.wikimedia.org/r/265179 (https://phabricator.wikimedia.org/T93735) (owner: 10Reedy) [06:55:57] (03CR) 10JanZerebecki: "Deployed to Jenkins: (['apps-android-wikipedia-periodic-test'])" [integration/config] - 10https://gerrit.wikimedia.org/r/264209 (owner: 10Niedzielski) [06:56:07] (03PS3) 10JanZerebecki: Update Android test target [integration/config] - 10https://gerrit.wikimedia.org/r/264209 (owner: 10Niedzielski) [06:56:19] (03CR) 10JanZerebecki: [C: 032] Update Android test target [integration/config] - 10https://gerrit.wikimedia.org/r/264209 (owner: 10Niedzielski) [06:58:31] (03Merged) 10jenkins-bot: Update Android test target [integration/config] - 10https://gerrit.wikimedia.org/r/264209 (owner: 10Niedzielski) [06:58:57] (03CR) 10JanZerebecki: WIP: Test MW-core against PHP55 for master/REL1_27+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [07:00:46] (03PS3) 10JanZerebecki: Preserve Android Spoon test output [integration/config] - 10https://gerrit.wikimedia.org/r/265009 (https://phabricator.wikimedia.org/T118100) (owner: 10Niedzielski) [07:09:50] (03CR) 10JanZerebecki: [C: 032] "Deployed to Jenkins: (['apps-android-wikipedia-periodic-test'])" [integration/config] - 10https://gerrit.wikimedia.org/r/265009 (https://phabricator.wikimedia.org/T118100) (owner: 10Niedzielski) [07:12:39] (03Merged) 10jenkins-bot: Preserve Android Spoon test output [integration/config] - 10https://gerrit.wikimedia.org/r/265009 (https://phabricator.wikimedia.org/T118100) (owner: 10Niedzielski) [07:14:00] (03PS2) 10JanZerebecki: Use Android Spoon test results [integration/config] - 10https://gerrit.wikimedia.org/r/266302 (owner: 10Niedzielski) [07:16:06] (03CR) 10JanZerebecki: [C: 032] "Deployed to Jenkins: (['apps-android-wikipedia-periodic-test'])" [integration/config] - 10https://gerrit.wikimedia.org/r/266302 (owner: 10Niedzielski) [07:19:30] (03Merged) 10jenkins-bot: Use Android Spoon test results [integration/config] - 10https://gerrit.wikimedia.org/r/266302 (owner: 10Niedzielski) [07:20:20] (03CR) 10JanZerebecki: [C: 04-1] 'recheck' on CR+2 now triggers gate-and-submit (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [07:26:02] (03PS2) 10JanZerebecki: Target testwiki for PdfHandler browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/266952 (owner: 10Dduvall) [07:38:57] !log reload zuul for 4951444..43a030b [07:39:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:44:34] (03CR) 10JanZerebecki: "Yes for these type of jobs the workspace may still be around for the next build, so your job needs to cope with a workspace as left from t" [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [07:46:06] (03CR) 10JanZerebecki: "But no that workspace is job specific, it is not share with any other job, just itself." [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [07:47:32] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-12-01, 3Wikidata-Sprint-2016-01-19: [Task] Add Wikidata to extension-gate in CI - https://phabricator.wikimedia.org/T96264#1973285 (10JanZerebecki) [07:47:46] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-12-01, 3Wikidata-Sprint-2016-01-19: [Task] Add Wikidata to extension-gate in CI - https://phabricator.wikimedia.org/T96264#1973286 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki [07:52:08] 10Continuous-Integration-Config, 10GeoData, 10Wikidata: Add GeoData to extension-gate in CI - https://phabricator.wikimedia.org/T125051#1973316 (10JanZerebecki) 3NEW [07:52:22] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050#1973325 (10JanZerebecki) [07:52:57] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050#1973302 (10JanZerebecki) [07:53:01] 10Continuous-Integration-Config, 10GeoData, 10Wikidata: Add GeoData to extension-gate in CI - https://phabricator.wikimedia.org/T125051#1973328 (10JanZerebecki) [08:33:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #858: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/858/ [09:18:44] (03CR) 10Zfilipin: [C: 032] Target testwiki for PdfHandler browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/266952 (owner: 10Dduvall) [09:26:13] (03Merged) 10jenkins-bot: Target testwiki for PdfHandler browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/266952 (owner: 10Dduvall) [09:35:09] 7Browser-Tests, 5Patch-For-Review: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1973478 (10Aklapper) [10:39:31] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1973598 (10Yurik) 3NEW a:3mobrovac [10:44:20] Project browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce build #1: 04FAILURE in 8.7 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce/1/ [10:45:20] Yippee, build fixed! [10:45:20] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #774: 09FIXED in 39 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/774/ [10:59:40] oh my god [10:59:49] PdfHadnler has been fixed somehow zeljkof ^^^ [11:00:09] hashar: I have rerun the job [11:00:15] it failed yesterday with a timeou [11:00:18] timeout [11:00:22] yeah random timeout indeed [11:00:38] I have deployed the test (not test2) job [11:00:58] but looks like jenkins does not want to get the latest commits from git :/ [11:01:11] https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce/ [11:06:17] (03PS1) 10Hashar: [CollaborationKit] generic jobs [integration/config] - 10https://gerrit.wikimedia.org/r/267011 [11:13:15] Project browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce-DEBUG build #1: 04FAILURE in 18 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce-DEBUG/1/ [11:14:31] (03CR) 10Hashar: [C: 032] [CollaborationKit] generic jobs [integration/config] - 10https://gerrit.wikimedia.org/r/267011 (owner: 10Hashar) [11:15:26] (03Merged) 10jenkins-bot: [CollaborationKit] generic jobs [integration/config] - 10https://gerrit.wikimedia.org/r/267011 (owner: 10Hashar) [11:18:55] zeljkof: I think that job is badly configured somehow [11:19:13] hashar: argh [11:19:25] there is a bunch of git related output at the top of https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce/3/consoleFull [11:19:35] where the Jenkisn git plugin poll the repo and attempt to determine which branch/commit to run [11:19:41] https://www.irccloud.com/pastebin/dbC6RgCq/ [11:19:46] apparently it has picked up 916d753d549b1ec65d4859db780696a4f118a56e [11:20:04] that is what it needs [11:20:16] which is indeed the tip of master [11:20:46] but take a look at files in workspace [11:20:48] https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce-DEBUG/ws/tests/browser/features/pdf.feature/*view*/ [11:21:04] https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce/3/git/ [11:21:04] it has "@test2.wikipedia.org" [11:21:04] origin/master: Build #3 of Revision 916d753d549b1ec65d4859db780696a4f118a56e (origin/master) [11:21:10] oh [11:21:10] instead of "@test.wikipedia.org" [11:21:39] and the job fails because it can not find any scenarios with "@test.wiki..." tag [11:21:55] and cucumber has: --tags @test.wikipedia.org --tags @firefox [11:22:15] yes [11:22:24] I am not sure what happened :/ [11:22:25] guess you want to flag the scenario for both test and test2 [11:22:55] no, we are moving to test [11:23:05] but the thing is, why did not files in workspace get updated? [11:23:20] to the latest commit in master [11:23:31] 😕 [11:27:02] zeljkof: look on the node that run the build maybe ? [11:27:27] hashar: hm [11:27:32] ran on integration-slave-trusty-1016.integration.eqiad.wmflabs [11:27:37] so, the node might be out of date [11:27:38] and [11:27:39] cd /mnt/jenkins-workspace/workspace/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce [11:27:41] forgot about that [11:28:13] commit 471fae17f6b4454c00a94b0d40cb92e8b492edd5 [11:28:13] Author: Dan Duvall [11:28:15] Date: Tue Jan 19 19:21:41 2016 +0000 [11:28:15] Creating new wmf/1.27.0-wmf.11 branch [11:28:19] that is the HEAD commit .... [11:28:26] $ git branch [11:28:26] * (detached from origin/wmf/1.27.0-wmf.11) [11:28:35] hm [11:28:41] how could that happen? [11:28:55] * hashar blames Jenkins git plugin [11:29:55] the tip of master is 916d753 [11:29:57] and it is properly tagged [11:30:11] but the HEAD of the local workspace is the wmf/1.27.0-wmf.11 branch [11:30:38] hashar: how do we fix it? [11:30:55] and it did git checkout -f 916d753d549b1ec65d4859db780696a4f118a56e [11:33:11] you did it manually? [11:33:25] I wanted to ask, how do we fix problems like this in the future? [11:33:36] report the bug upstream? [11:33:41] move away from jenkins? [11:35:06] well [11:35:08] gotta investigate [11:35:10] see what happens [11:40:22] well [11:40:33] zeljkof: lets nuke the workspace :-} [11:41:41] ahah same [11:41:47] yes [11:41:50] strange [11:46:26] (03CR) 10Zfilipin: "The new job is deployed, but failing. :(" [integration/config] - 10https://gerrit.wikimedia.org/r/266952 (owner: 10Dduvall) [11:47:11] (03CR) 10Zfilipin: "Do not delete the old job until the new one is fixed." [integration/config] - 10https://gerrit.wikimedia.org/r/266952 (owner: 10Dduvall) [11:47:15] zeljkof: I am replaying the commands [11:49:48] OHHHHHHHHHHHH [11:49:55] I AM **THAT** STUPID [11:49:56] of course [11:50:00] it is so fucking obvious [11:50:01] sorry [11:50:05] swearing in public [11:50:09] zeljkof: so [11:50:15] the git plugin works fine [11:50:16] but [11:50:32] the job runs against production [11:50:42] if you look at the relative timestamp 00:00:01.675 [11:50:55] what we do is invoke slave-scripts/bin/mw-api-siteinfo.py [11:51:02] which query the target site mediawiki API and ask for the remote branch [11:51:08] and it get wmf/1.27.0-wmf.11 [11:51:15] which is what is being deployed right now [11:51:25] then it does a git checkout -f origin/wmf/1.27.0-wmf.11 [11:51:25] Previous HEAD position was 916d753... QA: Target testwiki instead of test2wiki [11:51:29] and [11:51:43] the change merged in 'master' has not been backported to 'wmf/1.27.0-wmf.11' [11:51:44] :-} [11:51:55] extremely confusing stuff [11:54:46] :) [11:54:46] ah [11:55:05] I knew that too, and completely forgot :/ [12:12:07] ok, reproduced MultimediaViewer error locall [12:12:10] locally [12:14:36] great [12:14:55] seems test_name(foo) might just need to be replaced with foo.title [12:21:50] hashar: can you please add comment here? https://phabricator.wikimedia.org/T125072 [12:22:21] I have to finish something else first, so I will probably not have the time to fix it today [12:22:35] still catching up :( [12:23:09] zeljkof: just need to backport your patch to the wmf branch [12:23:14] it is just a cherrypick in gerrit :-} [12:23:39] hashar: which patch? [12:23:50] move from test2 to test for pdfhandler? [12:24:22] this? https://gerrit.wikimedia.org/r/#/c/266947/ [12:24:35] oh sorry [12:24:44] I though you were speaking of the test / wmf branch stuff [12:24:56] argh, I never know which branch to cherry pick to [12:25:35] which branch should I cherry pick it to? [12:26:13] what ever branch test.wikipedia.org is running [12:26:27] from https://test.wikipedia.org/wiki/Special:Version [12:26:36] 1.27.0-wmf.11 (ac4f4f2) [12:26:36] 21:34, 22 January 2016 [12:26:55] so if you cherry pick it to wmf/1.27.0-wmf.11 that should be fine ;} [12:27:57] hashar: like this? https://gerrit.wikimedia.org/r/#/c/267015/ [12:28:25] * zeljkof is out to lunch [12:28:34] feel free to merge the commit [12:28:37] the job should pass then [12:30:36] lunch time for me as well [12:38:05] 6Release-Engineering-Team, 10DBA, 10MediaWiki-Configuration, 6operations, and 2 others: codfw is in read only according to mediawiki - https://phabricator.wikimedia.org/T124795#1973722 (10jcrespo) @Joe, it is a configuration change. Should be "trivial". I opened this ticket because it has more repercussion... [13:09:48] Yippee, build fixed! [13:09:49] Project browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce build #8: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test.wikipedia.org-linux-firefox-sauce/8/ [13:10:45] hashar: ^ green [13:10:55] :-} [14:20:18] zeljkof: have you figured out the unknown test_name method ? https://phabricator.wikimedia.org/T125072 [14:20:38] hashar: sorry, no, working on something else at the moment [14:28:59] Project beta-scap-eqiad build #87932: 04FAILURE in 7 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/87932/ [14:37:10] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974014 (10hashar) [14:46:30] Yippee, build fixed! [14:46:31] Project beta-scap-eqiad build #87933: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/87933/ [14:48:00] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974035 (10hashar) I have updated the task details to show the git deploy status and the list of minions registered in Trebuchet redis server. The Trebuchet target are... [14:51:57] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974042 (10hashar) And the saltstack grain `deployment_target` seems to be properly setup: ``` root@deployment-salt:~ # salt 'deployment-sca0*' grains.get 'deployment_t... [14:55:34] !log Deleted salt keys deployment-pdf01.eqiad.wmflabs and deployment-memc04.eqiad.wmflabs (obsolete, entries with '.deployment-prep.' are already there) [14:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:58:03] oh my [14:58:17] Shinken is SOO useless [14:59:01] !log fixing puppet hostnames on deployment-memc02 [14:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:59:34] !log fixing puppet hostnames on deployment-elastic07 [14:59:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:00:44] !log Running puppet on deployment-memc02 and deployment-elastic07 . It is catching up with lot of changes [15:00:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:01:02] 5Testing-Initiative-2015, 10Browser-Tests-Infrastructure, 7JavaScript, 5Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#1974061 (10zeljkofilipin) [15:06:54] !log fixing puppet.con hostname on deployment-upload.deployment-prep.eqiad.wmflabs and running puppet [15:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:09:31] !log fixing puppet.conf hostname on deployment-upload deployment-conftool deployment-tmh01 deployment-zookeeper01 and deployment-urldownloader [15:09:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:20:39] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974106 (10hashar) On deployment-sca01 in `/var/log/salt/minion`: ``` 2016-01-28 15:09:45,290 [salt.minion ][ERROR ] The return failed for job 201601281509446622... [15:25:08] !log apt-get upgrade deployment-sca01 and deployment-sca02 [15:25:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:38:58] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974170 (10hashar) On sca01: ``` # ls -l /var/cache/salt/minion/extmods/returners/deploy_redis.py* -rw-r--r-- 1 root root 2947 Jan 28 01:43 /var/cache/salt/minion/extmo... [15:55:29] 10Beta-Cluster-Infrastructure, 6Labs, 6operations: Duplicate IP address DNS entry - https://phabricator.wikimedia.org/T125040#1974191 (10hashar) integration-t102108-jessie-new2 is an old one that got deleted but apparently is still registered in LDAP :( [15:56:14] legoktm: Could i have some help with extension registration. [16:03:10] PROBLEM - Puppet failure on deployment-cache-parsoid05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:40] (03PS5) 10Cdentinger: WIP: Adds DonationInterface test against mediawiki 1.25 [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) [17:15:35] why are npm jobs shitting themselves so badly today? :( [17:15:39] e.g. https://integration.wikimedia.org/ci/job/npm-run-demos/1344/console [17:15:58] https://gerrit.wikimedia.org/r/#/c/267026/ just failed to merge three times in a row, on a different npm job every time [17:16:07] help :( [17:22:18] 10Deployment-Systems, 6Release-Engineering-Team, 7WorkType-NewFunctionality: Use subrepos instead of git submodules for deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#1974475 (10mmodell) p:5High>3Normal [17:24:17] 10Deployment-Systems, 3Scap3, 7WorkType-NewFunctionality: default lock file for scap3 should be repo-dependent - https://phabricator.wikimedia.org/T116208#1974487 (10mmodell) 5Open>3Resolved [17:25:47] 10Deployment-Systems, 3Scap3: sync-masters slow (on mira?) - https://phabricator.wikimedia.org/T125108#1974494 (10greg) 3NEW [17:28:33] 10Deployment-Systems, 3Scap3: sync-masters slow (on mira?) - https://phabricator.wikimedia.org/T125108#1974514 (10Krenair) Whereas on tin: ```17:27:49 Started sync-masters sync-masters: 100% (ok: 1; fail: 0; left: 0) 17:28:09 Finished sync-masters (duration: 00m 20s)``` [17:31:32] 10Beta-Cluster-Infrastructure: broken Upload file link in en-beta navigation - https://phabricator.wikimedia.org/T59117#1974529 (10Luke081515) a:3Luke081515 [17:33:01] Hey. Someone needs to run `npm cache clean` on the CI infrastructure. Should I file a ticket? [17:35:39] marxarelli: thcipriani zeljkof hasharAway ^ [17:36:34] James_F: I can certainly run npm cache clean somewhere. [17:36:43] 10Beta-Cluster-Infrastructure, 6Labs, 6operations: Duplicate IP address DNS entry - https://phabricator.wikimedia.org/T125040#1974545 (10Andrew) 5Open>3Resolved a:3Andrew It's in designate, not ldap. But, yes, looks like we leaked one; I've cleaned it up. [17:37:01] thcipriani: :-) Do you know which machines in CI are the ones that act as the npm caches? Do they act as their own caches? [17:37:37] James_F: is there a failure you can link us to so we can better troubleshoot? [17:38:14] James_F: I don't know off the top of my head. There does seem to be an npm cache in /mnt/home/jenkins-deploy, FWIW [17:39:18] marxarelli: https://integration.wikimedia.org/ci/job/npm-run-demos/1344/console for instance. [17:39:37] James_F: rad. ty [17:41:20] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#1974574 (10Luke081515) [17:41:22] 10Beta-Cluster-Infrastructure: broken Upload file link in en-beta navigation - https://phabricator.wikimedia.org/T59117#1974572 (10Luke081515) 5Open>3Resolved Seems like it works now, after import. Feel free to reopen, if something does not. [17:54:14] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1974658 (10Andrew) 5duplicate>3Open I'm re-opening... there are several issues in a row that CI is rejecting. And of course fixing thos... [17:55:05] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing - https://phabricator.wikimedia.org/T125067#1974668 (10mobrovac) This seems to be an issue related to reporting, because the code is actually deployed on those servers. [17:56:09] 10Continuous-Integration-Config, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#1974691 (10Andrew) Surely the CI environment can stub out calls and make those ldap calls return a re... [18:01:00] sorry wikibugs [18:01:25] * Luke081515 should use smaller bulk edits next time [18:03:55] Luke081515: huh, that should have been fixed [18:04:08] Yeah, but my bulk edit was real big [18:04:24] greg-g: Why does wikibugs notify channels if a suscriber changes? [18:07:11] Luke081515: herald was doing other things, eg: [18:07:12] Luke081515 removed a subscriber: wikibugs-l-list. [18:07:12] Herald added subscribers: Steinsplitter, Matanya. [18:07:12] Herald added a project: Commons. [18:07:20] hm, ok [18:07:35] that's from an email notif I just got [18:07:48] herald doesn't act on things until there's an action taking place [18:08:02] (I said that badly, but I think you know what I mean) [18:08:05] I would not submit new jobs, I think, but first I have to wait till the current job finishes... [18:08:16] yeah [18:08:22] :) [18:09:10] It may be useful to ask upstream for a button to stop a bulk edit, or pause him. What do you think? [18:09:37] * greg-g shrugs [18:09:51] Luke081515: how big was it? [18:10:02] really big [18:10:12] like? :) [18:10:14] the problem is, there are a lot of tasks [18:10:15] 1000 [18:10:15] 100 or 1000? [18:10:17] :) [18:10:45] At first, I want to take 5000, but I realized that this is too big [18:10:56] I would choose small ones next time [18:12:21] !log running npm cache clean on integration-slave-precise-1011 sudo -i -u jenkins-deploy -- npm cache clean [18:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:16:43] bulk edits are restriced as far i know and herad is automated stuff. [18:16:58] greg-g^^ [18:17:01] Steinsplitter: yes, I know [18:17:18] !log cleaning npm cache on slave machines: salt -v '*slave*' cmd.run 'sudo -i -u jenkins-deploy -- npm cache clean' [18:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:17:31] herald looks at all transactions currently happening, sees if they match a rule, then does something. [18:20:06] marxarelli: did you get the npm cache / workspace pruned ? [18:21:22] hashar: yeah, thcipriani is fixing it [18:21:25] hashar: working on it now, cleared integration-slave-precise-1011 with npm cache clean, retrying a job, verifying, then running on all slaves... [18:21:41] sounds good [18:21:52] I wanted to migrate the npm jobs to nodepool instances [18:22:02] but puppet fails to install npm/node on Jessie :-((( [18:22:10] back to dinner [18:22:15] usually, I salt delete the workspaces [18:22:25] unless they get mediawiki/core since it takes 3 - 4 minutes to clone it [18:22:31] foood [18:23:26] * marxarelli thinks it's a crime to be in France and not be perpetually eating [18:39:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1978973 (10scfc) When I look at https://integration.wikimedia.org/ci/job/mwext-testextension-php53/941/consoleText, the only two errors that... [18:44:27] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing / salt problems in Beta - https://phabricator.wikimedia.org/T125067#1978999 (10mobrovac) a:5mobrovac>3None [18:44:41] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing / salt problems in Beta - https://phabricator.wikimedia.org/T125067#1973598 (10mobrovac) p:5Triage>3Normal [18:45:29] marxarelli: actually, it's considered a crime not to eat at noon in France :) [18:46:12] <3 France .... snaaaaaaacks [18:47:06] thcipriani: jsut nuking ~/.npm and the global cache is usually a better approach than npm cache clean [18:47:39] thcipriani: btw, when you have time, could you look into https://phabricator.wikimedia.org/T125067 ? trebuchet and salt seem to be having some fun in beta [18:48:42] mobrovac: oh boy. Sure I can take a look. [18:49:25] thnx [18:49:30] __pillar__ not defined does sound bad. [18:51:26] yup [18:51:29] it does [18:51:30] :) [19:54:12] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1979246 (10ori) 3NEW [20:03:57] ostriches: thcipriani: twentyafterfour: is one of you familiar with trebuchet / salt ? [20:04:12] yurik can't deploy Graphoid on beta cluster [20:04:24] its targeting jessie boxes sca01/sca02 [20:04:53] hashar: I am barely [20:04:54] and the salt_call invoking extmods/returners/deploy_redis.py fails with NameError: global name '__pillar__' is not defined. [20:04:56] :( [20:05:02] the task has all the detail https://phabricator.wikimedia.org/T125067 [20:05:17] I did the basic first level forensic, but i have no f*** clue what is going on [20:05:35] thcipriani: or ostriches probably know it better than I do [20:06:01] actually looks like thcipriani was already looking into the same issue (see above) [20:11:15] hashar: yeah, I was just looking into that, actually. I think it has to do with the returner. [20:12:18] although some preliminary testing seems like everything is working there. I was just about to dust off some redis commands to troubleshoot. [20:13:17] :-/ [20:13:36] thcipriani: I am pretty sure the repo was not checked out on the minions / deployment target [20:13:40] but might be wrong [20:13:46] can't remember whether I reported back on the task [20:14:05] thcipriani: also, the deployment-sca0x got switched to Jessie [20:14:08] to get nodejs4.2 [20:14:13] so that might be a side effect [20:14:51] hashar: hmm...the local deploy.fetch seemed to work just fine. Return code of 0. I'll dig some more and update the ticket with what I find. [20:15:16] maybe it magically fixed :D [20:15:32] there were definitely some problems earlier this week [20:16:02] redis on deployment-bastion tried to became a slave of mira :) [20:16:26] and everything was reporting "0/X minions completed [stage]" [20:17:17] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1979333 (10mmodell) [20:17:19] nothing about __pillar__ missing. Also: sudo salt-call pillar.items seemed fine on that sca01, so not entirely sure why pillar went away... [20:17:47] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1979336 (10mmodell) @ori: Does {D111} look about right? [20:18:05] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:17] twentyafterfour: Other option is moving to python 3.5 and we'd get it for free with pep 475 :P [20:18:36] ostriches: yeah, I would really like to move to python 3.5 but that's not gonna happen [20:19:00] unless there is a backport and we can talk ops into letting us install it [20:19:43] 3.4 is what I've got an I'm running a rather modern ubuntu (15.04, I guess it isn't that new now) [20:22:57] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39497 bytes in 0.623 second response time [20:29:16] ostriches: twentyafterfour pep are sometime backported to previous minor versions [20:29:34] I doubt 475 will be :p [20:34:00] 6Release-Engineering-Team: Blockers to deploying 1.27-wmf.11 - https://phabricator.wikimedia.org/T125143#1979397 (10greg) 3NEW a:3dduvall [20:37:39] greg-g: Does this mean, that group 0/1 get revertet to WMF 10? [20:39:11] no [20:39:22] 0 and 1 are on wmf.11, 2 is on wmf.10 [20:39:38] we're determining if we should go out to group2 today or not :/ [20:39:45] (again, jus tlike last week) [20:43:21] The task looks interesting... one security issue there.... [20:43:30] or a restricted issue [20:44:21] yeah, restricted just because it has session info in it [20:44:35] ah, ok [20:46:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:16] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:37] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing / salt problems in Beta - https://phabricator.wikimedia.org/T125067#1979476 (10thcipriani) I first tried a fetch of the repo on `deployment-sca01`, which seemed just fine: ```thcipriani@deployment-sca01:~$ sudo... [20:54:06] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39834 bytes in 1.077 second response time [20:56:28] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39508 bytes in 1.357 second response time [21:02:10] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing / salt problems in Beta - https://phabricator.wikimedia.org/T125067#1979516 (10thcipriani) So it seems that the redis `fetch_tag` is not being updated, even thought the `fetch_checkin_timestamp` gets updated on d... [21:13:47] 6Release-Engineering-Team: Blockers to deploying 1.27-wmf.11 - https://phabricator.wikimedia.org/T125143#1979553 (10greg) [21:14:50] 6Release-Engineering-Team: Blockers to deploying 1.27-wmf.11 - https://phabricator.wikimedia.org/T125143#1979397 (10greg) per the update in the restricted task, all of the blockers have been addressed (they are being kept open until we verify everything after wmf.11 hits wikipedias). The non-blockers are still... [21:16:55] twentyafterfour: yo. Ori can't push to PHSTAGE. All users should be able to tho :/ [21:17:07] ahoy [21:17:30] 21:16 <+ostriches> twentyafterfour: yo. Ori can't push to PHSTAGE. All users should be able to tho :/ [21:18:00] fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/PHSTAGE/harbormaster-staging.git/': The requested URL returned error: 403 / STAGING FAILED Unable to push changes to the staging area. [21:18:02] D112 shows the failure [21:18:39] Did it prompt for a pass or did you use .netrc or something? [21:19:11] it prompted me for a pass [21:19:24] Ah. Did you use your normal password? [21:19:31] but it must have worked, because apart from pushing to PHSTAGE, it did create the diff [21:19:34] yes [21:19:47] Vcs password a la gerrit [21:19:53] See your settings [21:20:47] The diff succeeded because it wasn't doing git...it was using your API token [21:21:20] got it. what should i run now, to retry? [21:22:19] `arc diff --update` should do it, although it'll probably complain there was no change. [21:22:48] I dunno how to just push the staging bit and retrigger harbormaster [21:23:01] hahaha, there's arc diff --excuse [21:23:03] i love phab [21:24:03] :) [21:24:31] it failed again, though maybe i have the wrong password [21:24:33] i'll try to reset it [21:24:54] You're looking at https://phabricator.wikimedia.org/settings/panel/vcspassword/, right? [21:25:12] yeah [21:25:50] i tried resetting and it still fails [21:26:06] is there something i can do to try out the vcs auth without creating spammy changes? [21:26:35] Erm... [21:27:08] Other option, if you know your ssh works, is to rewrite the URL in your gitconfig [21:27:27] ok, it worked [21:27:35] whee \o/ [21:27:35] i was using my ldap username with the vcs password [21:27:44] needed to use the attached phab account username instead [21:28:01] thanks folks [21:28:07] Ahhh, yeah the CN from LDAP is only used for logging in if you're /actually using ldap to login/ [21:28:09] Nothing else :) [21:28:44] wee, and it passes now too [21:34:17] (03PS1) 10Hashar: [OpenStackManager] depends on LdapAuthentication [integration/config] - 10https://gerrit.wikimedia.org/r/267147 (https://phabricator.wikimedia.org/T124613) [21:36:26] (03CR) 10Hashar: [C: 032] [OpenStackManager] depends on LdapAuthentication [integration/config] - 10https://gerrit.wikimedia.org/r/267147 (https://phabricator.wikimedia.org/T124613) (owner: 10Hashar) [21:36:52] (03CR) 10Andrew Bogott: [C: 031] "Not 100% clear to me that this will fix the problem, but it seems correct in any case." [integration/config] - 10https://gerrit.wikimedia.org/r/267147 (https://phabricator.wikimedia.org/T124613) (owner: 10Hashar) [21:37:21] (03Merged) 10jenkins-bot: [OpenStackManager] depends on LdapAuthentication [integration/config] - 10https://gerrit.wikimedia.org/r/267147 (https://phabricator.wikimedia.org/T124613) (owner: 10Hashar) [22:21:37] 6Release-Engineering-Team: Blockers to deploying 1.27-wmf.11 - https://phabricator.wikimedia.org/T125143#1979821 (10greg) [22:23:19] !log salt '*slave-precise*' cmd.run 'apt-get install php5-ldap' ( https://phabricator.wikimedia.org/T124613 ) will need to be puppetized [22:23:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:23:23] 6Release-Engineering-Team: Blockers to deploying 1.27-wmf.11 - https://phabricator.wikimedia.org/T125143#1979845 (10Jdforrester-WMF) [22:25:56] 10Continuous-Integration-Config, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#1979874 (10hashar) So to get it fixed we need: [x] LdapAuthentication injected as a dependency by CI... [22:27:21] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI slaves need package php5-ldap for OpenStackManager/LdapAuthentication - https://phabricator.wikimedia.org/T125158#1979880 (10hashar) 3NEW [22:31:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: CI slaves need package php5-ldap for OpenStackManager/LdapAuthentication - https://phabricator.wikimedia.org/T125158#1979909 (10hashar) a:3hashar [22:31:29] 10Continuous-Integration-Config, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#1979911 (10hashar) a:3hashar [22:38:23] 7Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: Fix failed MultimediaViewer browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94157#1979949 (10Jdlrobson) Is anyone monitoring the failing Firefox, IE8 and IE9 builds? I'd be tempted to kill them and just go with the passing https://integrat... [22:41:19] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-OpenStackManager: CI rejecting every patch to OpenStackManager - https://phabricator.wikimedia.org/T125008#1979972 (10Andrew) 5Open>3Resolved @scfc, you're right. And, in any case, this is fixed! [22:41:30] 10Continuous-Integration-Config, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#1979974 (10Andrew) 5Open>3Resolved [22:47:08] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1980008 (10mmodell) ok nevermind D111 [22:49:17] ostriches / ori: fwiw, the staging repo is only used by jenkins so it's not 100% critical [22:54:00] 7Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: Fix failed MultimediaViewer browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94157#1980062 (10Gilles) And destroying man-weeks of work out of laziness to actually address a minor upgrade issue. Can you please truly assume ownership of tha... [23:22:48] !log Updated portals on betalabs to master [23:22:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:23:09] cluster* cluster* cluster* [23:23:14] * greg-g twitches [23:48:10] (03PS6) 10Cdentinger: Adds DonationInterface test against mediawiki 1.25 [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) [23:59:11] for the record, I yet again really appreciate bd808 sal tool [23:59:20] 's*