[00:09:45] PROBLEM - Puppet staleness on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [43200.0] [00:15:11] Krinkle: Did the fix for the lib/ve (submodule) jenkins upgrade thing break? https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/8649/console etc. [00:15:29] (Nothing obvious in the RelEng log.) [00:30:41] James_F: I worked around it for now [00:30:42] Should be fine [00:30:51] Krinkle: Thanks. [00:30:51] Fixed a few days back [00:33:30] Krinkle: Oh. So… we're just stuffed? [00:34:05] Krinkle: My link was to it happening 30 minutes ago. A re-regression. [00:34:10] James_F: Right [00:34:29] James_F: My work around fixed the fact that new builds were not checking out the relevant submodule commit [00:34:36] but it probably doesn't account for the submodule changing [00:34:43] previously it was stuck on some random wmf6 commit [00:34:45] I fixed that [00:34:49] let me see [00:35:14] Krinkle: Ah. :-( [00:35:27] Krinkle: I thought all that was needed was recompiling the JJB jobs? [00:36:31] James_F: https://phabricator.wikimedia.org/T101105 [00:36:33] Yeah, it should [00:36:40] My fix was before Antoine refreshed the jobs [00:36:50] live hack for ve-npm [00:37:01] So… the refresh broke your fix and didn't fix the thing it was fixing? :-( [00:37:07] that is now undone, which makes sense but the refresh should've been a no-op [00:37:27] https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/ [00:37:31] It only started at 3pm though [00:37:40] Ah, which is exactly when it was refreshed [00:37:47] the gear icon shows when the reconfigure happens [00:39:03] So the fix only works until we update VE-core? :-) [00:39:37] No, it failed before that as well [00:39:47] I guess he didn't test his change before updating 500 jobs [00:39:49] :( [00:39:54] Ah. :-( [00:39:56] (or after) [00:45:47] Ugh, actually. No, the fix described in that task by updating JJB probably works [00:46:03] but that's not what was deployed. The deployed fix in the job config diff actually does the opposite [00:46:57] Helpful. [00:52:13] Krinkle: You fixed with a one-off hack again or fixed for good? [00:53:03] James_F: I updated the job to use the newer git config [00:53:07] that's indefinte [00:53:13] Aha, good. [00:53:16] Like I did on monday (or whenever it was) [00:53:19] Until we next push the job? [00:53:26] Yeah, it's a live hack. [00:53:56] The wide scale job regeneration should've bumped the version but instead it seemed to have done a no-op on most jobs putting it to the old version, and in case of ve back to was it was before [00:54:03] Is the hack something we can make in git and merge as "already pushed" so next time it's not broken? [00:54:11] No [00:54:13] Ah. [00:54:26] It's internal data in the job config xml file. [00:54:36] JJB generates it correctly when using a newer JJB version to generate the XML. [00:54:39] It's not visible in git. [00:54:43] (which are yaml files) [00:55:15] Eurgh. [00:55:17] Great. [00:55:46] Our config is more or less the same. The version update is handled by JJB. Which is great that they keep compat, but it means the update is on the deployer side (the conversion tool, jjb), not the config itself. [00:56:06] The change is already merged in git/integration/config [00:56:27] But needs to be deployed from local machine to jenkins using a newer jjb version. Which I thought Antoine did, since he told me about that. [00:56:33] Oh well, we're good now for VE. [00:56:41] :-( [00:56:44] Thanks! [02:07:13] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL 100.00% of data above the critical threshold [43200.0] [02:58:54] Yippee, build fixed! [02:58:55] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #707: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/707/ [03:39:26] Yippee, build fixed! [03:39:27] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #669: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/669/ [03:56:29] Yippee, build fixed! [03:56:29] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #144: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/144/ [06:49:35] (03CR) 10Polybuildr: "Addshore, any updates on this one? It's been waiting for love from April Fools' Day." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/153399 (owner: 10Addshore) [07:06:45] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:21:01] Yippee, build fixed! [07:21:01] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #463: FIXED in 56 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/463/ [08:18:39] PROBLEM - Puppet staleness on deployment-videoscaler01 is CRITICAL 30.00% of data above the critical threshold [43200.0] [08:52:25] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 55.56% of data above the critical threshold [0.0] [09:25:06] (03PS1) 10Polybuildr: Enable npm test for SmiteSpam extension [integration/config] - 10https://gerrit.wikimedia.org/r/215870 (https://phabricator.wikimedia.org/T101374) [09:25:16] Krinkle: Can I please get a +2 on that ^ [09:26:19] (03CR) 10Krinkle: [C: 032] Enable npm test for SmiteSpam extension [integration/config] - 10https://gerrit.wikimedia.org/r/215870 (https://phabricator.wikimedia.org/T101374) (owner: 10Polybuildr) [09:26:31] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/215870 [09:26:37] Logged the message, Master [09:27:12] Krinkle: Thanks again! :) [09:27:56] (03Merged) 10jenkins-bot: Enable npm test for SmiteSpam extension [integration/config] - 10https://gerrit.wikimedia.org/r/215870 (https://phabricator.wikimedia.org/T101374) (owner: 10Polybuildr) [09:47:00] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #505: FAILURE in 9 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/505/ [09:47:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL 20.00% of data above the critical threshold [0.0] [10:02:47] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [10:04:11] Yippee, build fixed! [10:04:12] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #452: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/452/ [10:17:27] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [10:23:56] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [10:24:38] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [10:25:34] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [10:26:06] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [10:27:36] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [10:28:06] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [10:28:40] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [10:29:08] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [10:30:04] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [10:31:04] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [10:31:27] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [10:32:37] RECOVERY - Puppet failure on deployment-logstash1 is OK Less than 1.00% above the threshold [0.0] [10:33:09] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [10:44:03] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [10:52:33] (03PS1) 10Hashar: Switch Wikibase to the generic npm job [integration/config] - 10https://gerrit.wikimedia.org/r/215875 (https://phabricator.wikimedia.org/T101105) [10:52:35] (03PS1) 10Hashar: Switch WikidataQuality to the generic npm job [integration/config] - 10https://gerrit.wikimedia.org/r/215876 (https://phabricator.wikimedia.org/T101105) [11:00:12] (03CR) 10Hashar: [C: 032] Switch Wikibase to the generic npm job [integration/config] - 10https://gerrit.wikimedia.org/r/215875 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [11:02:11] (03Merged) 10jenkins-bot: Switch Wikibase to the generic npm job [integration/config] - 10https://gerrit.wikimedia.org/r/215875 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [11:04:43] (03CR) 10Hashar: [C: 032] "Wikibase worked :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/215876 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [11:06:38] (03Merged) 10jenkins-bot: Switch WikidataQuality to the generic npm job [integration/config] - 10https://gerrit.wikimedia.org/r/215876 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [11:26:34] (03PS1) 10Hashar: Zend Wikibase jobs only triggered in gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) [11:27:24] (03CR) 10Hashar: [C: 04-1] "Pending Wikidata / Wikibase devs approval either here or on T101382" [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) (owner: 10Hashar) [11:57:42] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [11:58:02] hashar: the jjb update failed. It reverted all jobs to be broken instead of fixing them. [11:58:12] (a no-op for most of them, a regression in VE, again) [11:58:14] :-( [11:58:27] yeah I am looking at it [11:58:29] Did you test any of the jobs with submodules before or after deplot? [11:58:59] the git-remoteonly-zuul scm macro does not have a submodule: section [11:59:05] so JJB ends up generating the old configuration :-((( [11:59:28] I can't find out whether git-remoteonly-zuul should have submodules or not [12:00:50] I guess I will just explicitly set submodule: [12:00:52] hashar: it should have submodules. If you look in default.yaml, the description does not say that it "does NOT" have submodules, and the default is yes. [12:01:03] the git plugin used to have it by default and disable via the option [12:01:11] the new plugin doesn't load the submodule options by default [12:01:22] but if any of the options are loaded, then it will still enable submodule by default [12:01:28] but if the section is missing, it defaults to not. [12:08:02] Krinkle: so it seems the new git plugin default to not processing submodules [12:10:26] yeah [12:10:34] default behavior changed :-( [12:10:35] https://integration.wikimedia.org/ci/job/test-hashar-submodule/1/console [12:10:40] that clones VE extension [12:12:49] (03PS1) 10Hashar: Force JJB to use new git submodule syntax [integration/config] - 10https://gerrit.wikimedia.org/r/215888 (https://phabricator.wikimedia.org/T101105) [12:13:23] Krinkle: ^^^ [12:13:26] I am refreshing the job [12:13:47] diff https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/jobConfigHistory/showDiffFiles?timestamp1=2015-06-04_00-47-19×tamp2=2015-06-04_12-13-24 [12:17:07] yeah, that diff is a no-op. It leaves the live hack I did last night as-is. Okay :) [12:17:21] the false inside SubmoduleOptin [12:17:42] that is really a mess [12:19:54] (03CR) 10Hashar: [C: 032] Force JJB to use new git submodule syntax [integration/config] - 10https://gerrit.wikimedia.org/r/215888 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [12:20:03] will have to fix it up properly eventually [12:20:11] (03PS1) 10Hashar: Force JJB to use new git submodule syntax (2) [integration/config] - 10https://gerrit.wikimedia.org/r/215893 (https://phabricator.wikimedia.org/T101105) [12:21:55] (03Merged) 10jenkins-bot: Force JJB to use new git submodule syntax [integration/config] - 10https://gerrit.wikimedia.org/r/215888 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [12:23:20] (03CR) 10Hashar: [C: 032] Force JJB to use new git submodule syntax (2) [integration/config] - 10https://gerrit.wikimedia.org/r/215893 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [12:23:24] Krinkle: sorry for the mess :-((((( [12:24:29] No worries. I'm glad we figured it out. [12:24:49] It wasn't a documented breaking change [12:24:51] Meh [12:25:17] (03Merged) 10jenkins-bot: Force JJB to use new git submodule syntax (2) [integration/config] - 10https://gerrit.wikimedia.org/r/215893 (https://phabricator.wikimedia.org/T101105) (owner: 10Hashar) [12:27:23] Krinkle: in jjb that should probably have been made a git2 macro :-} [12:41:17] Krinkle: at least ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint' [12:41:18] [12:41:20] that should be fixed now [12:41:24] was https://phabricator.wikimedia.org/T93321 [12:41:28] and required a bump of JJB version [12:41:28] yeah [12:41:35] which is what triggered the git plugin upgrade [12:42:00] I tend to hates those chain of tasks [12:42:10] you have a easy A but that needs B which needs C which needs D etc [13:03:38] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #670: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/670/ [13:27:01] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [13:32:41] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [13:36:35] RECOVERY - Puppet failure on deployment-fluoride is OK Less than 1.00% above the threshold [0.0] [13:37:37] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [14:03:33] (03CR) 10JanZerebecki: [C: 031] "Talked to a few people and everyone thinks its a good idea." [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) (owner: 10Hashar) [14:12:32] jzerebecki: you are awesome :-} [14:12:38] (03PS2) 10Hashar: Zend Wikibase jobs only triggered in gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) [14:14:30] (03CR) 10Hashar: [C: 032] "Alea Jacta Est" [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) (owner: 10Hashar) [14:16:08] (03Merged) 10jenkins-bot: Zend Wikibase jobs only triggered in gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/215880 (https://phabricator.wikimedia.org/T101382) (owner: 10Hashar) [14:39:45] RECOVERY - Puppet staleness on integration-slave-jessie-1001 is OK Less than 1.00% above the threshold [3600.0] [14:54:19] !log integration-jessie-slave1001 : running dpkg --configure -a [14:54:28] Logged the message, Master [14:58:09] !running sudo salt '*' cmd.run 'sed -i "s/GlobalSign_CA.pem/ca-certificates.crt/" /etc/ldap/ldap.conf' on integration-saltmaster [14:58:16] !log running sudo salt '*' cmd.run 'sed -i "s/GlobalSign_CA.pem/ca-certificates.crt/" /etc/ldap/ldap.conf' on integration-saltmaster [14:58:20] Logged the message, Master [15:00:23] legoktm: Could I grab your attention at some point on twentyafterfour's comments on https://gerrit.wikimedia.org/r/#/c/214661/ ? I'd really love to get it landed so we can run ReleaseTaggerBot on extensions… [15:07:14] !log integration-jessie-slave1001 : upgrading salt from 2014.1.13 to 2014.7.5 [15:07:19] Logged the message, Master [15:14:15] RECOVERY - Puppet failure on integration-vmbuilder-trusty is OK Less than 1.00% above the threshold [0.0] [15:22:16] PROBLEM - Puppet failure on integration-slave-trusty-1021 is CRITICAL 55.56% of data above the critical threshold [0.0] [15:31:33] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK Less than 1.00% above the threshold [0.0] [15:33:17] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK Less than 1.00% above the threshold [0.0] [15:40:03] RECOVERY - Puppet failure on integration-publisher is OK Less than 1.00% above the threshold [0.0] [15:47:12] RECOVERY - Puppet failure on integration-slave-trusty-1021 is OK Less than 1.00% above the threshold [0.0] [15:50:15] 10Deployment-Systems, 6Release-Engineering, 7Epic, 3releng-201415-Q4: EPIC: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1337816 (10greg) [15:59:10] twentyafterfour: on your patch management tool, I will totally +2 it if you fix the php coding style issues I pointed out. I don't mean blatant disrespect but I don't care what the opinion of the ops team is about the tools that used to manage releases and deployments that they don't have anything to do with. [15:59:57] :) [16:00:08] If you and ostriches and thcipriani want a tool then you should have it [16:03:14] mwpatch is twentyafterfour's baby [16:03:26] No objections or opinions here lol [16:04:10] bd808: thanks :) [16:05:43] I don't feel as strongly about the code as I do about exactly what bd808 just said. [16:05:48] I continue to have disbelief in the concept of once-size-fits-most deployment tooling. And ffs the ops team blocked me from using fabric to rebuild scap [16:06:07] :( [16:06:57] really? what was the objection to that? sounds to me like "reinventing a wheel" lol [16:07:33] they wanted me to get rid of scap all together and deploy using trebuchet on the basis of "one tool is better than two" [16:07:59] one tool is better than two but only if that tool works for what you are doing ;) [16:08:14] and one isn't better than two if the one does both jobs badly [16:08:15] :) it was political at that point [16:09:27] the unholy union of puppet and salt is what kept me from it in the longer term [16:09:44] that should be made into a contest - win $1m if you manage to deploy MW via trebuchet [16:10:09] I figured out how to replace scap with trebuchet and have notes on a complete plan somewhere but it requires all deployers to have root because salt if so flakey that you can't use it without god powers [16:10:29] *is so flakey [16:10:54] this may be the fault of our salt configuration and not the tool itself but I have no real way to evaluate that [16:11:55] I do know that I used mcollective to manage deployment orchestration includeing rolling elasticsearch cluster restarts with no issues at all but I can't get salt to ping consistently [16:13:47] le sigh. [16:14:23] thcipriani: it will all be fixed in the next realease! [16:14:47] I think determinism got pushed out a few releases :) [16:16:21] It's getting close to time to revisit this argument again, and sadly I'm just not comfortable with the answer being salt [16:17:39] My big thing is the mix of salt and puppet. it's just wrong. Ryan did that because he hates Puppet and wanted to kill it off [16:17:46] that so far has not happened at all [16:18:19] if we were 100% salt I bet it would work much better because we would really rely on it [16:18:44] I don't see puppet going away. but it seems like there is a lot of love for python in this org. I don't know how puppet got as established as it is really. I don't love puppet but I don't hate it. Salt should work better than it does. I don't get it. [16:18:49] mcollective is awesome [16:19:02] (not really awesome but it's relatively awesome) [16:20:03] With mcollective I could write really cool tools really quickly [16:20:08] manybubbles: https://gerrit.wikimedia.org/r/#/c/214821/ if not seen. [16:20:26] the api was straight-forward and easy to use. it was fairly reliable. And importantly, it was simple. [16:32:05] (03PS3) 10Legoktm: make-wmf-branch: Move list of branched extensions/skins to a JSON file [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) [16:32:16] (03CR) 10Legoktm: make-wmf-branch: Move list of branched extensions/skins to a JSON file (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) (owner: 10Legoktm) [16:33:18] legoktm: Thanks! [16:41:22] (03PS4) 10Legoktm: make-wmf-branch: Move list of branched extensions/skins to a JSON file [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) [16:41:43] greg-g: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=161758&oldid=161754 for you. :-) [16:41:52] (03CR) 10Legoktm: "PS3: Fixed local.conf override. PS4: Set up phpunit to lint the JSON file." [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) (owner: 10Legoktm) [16:42:34] twentyafterfour: let me know if/when you think ^ is ready, and I'll do the CI change first, and then you can +2 it to make sure the tests pass [16:44:33] legoktm: looks good to me [16:45:24] ok, doing the CI change now [16:47:58] (03PS1) 10Legoktm: Set up composer-test + jsonlint for mediawiki/tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/215935 [16:48:13] (03CR) 10Legoktm: [C: 032] Set up composer-test + jsonlint for mediawiki/tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/215935 (owner: 10Legoktm) [16:49:52] legoktm: ok so +2 it? [16:49:57] (03Merged) 10jenkins-bot: Set up composer-test + jsonlint for mediawiki/tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/215935 (owner: 10Legoktm) [16:50:12] !log deploying https://gerrit.wikimedia.org/r/215935 [16:50:16] Logged the message, Master [16:50:36] (03CR) 10Legoktm: "recheck" [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) (owner: 10Legoktm) [16:50:57] twentyafterfour: yep :) [16:52:43] (03CR) 1020after4: [C: 032] make-wmf-branch: Move list of branched extensions/skins to a JSON file [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) (owner: 10Legoktm) [16:52:54] (03Merged) 10jenkins-bot: make-wmf-branch: Move list of branched extensions/skins to a JSON file [tools/release] - 10https://gerrit.wikimedia.org/r/214661 (https://phabricator.wikimedia.org/T100817) (owner: 10Legoktm) [16:53:40] I'm gonna attempt to introduce a json error, just to test it out [16:56:58] 10Deployment-Systems, 6Release-Engineering, 5Patch-For-Review: Move list of extensions and skins branched in make-wmf-branch out of a PHP file - https://phabricator.wikimedia.org/T100817#1338066 (10Legoktm) 5Open>3Resolved [16:57:08] ok [16:57:21] to test it locally, just run "composer update && composer test" in the repo root [16:58:27] legoktm: ok [16:58:44] I was gonna try to commit and then abandon the error, but maybe not necessary? [16:58:53] I mean, push to gerrit [16:58:56] if you want to be extra confident :P [17:02:10] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338071 (10Ricordisamoa) >>! In T101358#1336865, @hashar wrote: > What about having the Phabfile template to use the callsign? It is more or less meant to be an... [17:03:56] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338077 (10Nemo_bis) > Getting things to speak callsigns What things? Callsigns are not meant to be usable or used. Deprecating the "project" parameter of https... [17:13:56] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338106 (10mmodell) If we stored the mapping in phabricator then a bit of custom code could look up the phabricator callsign based on the gerrit project. We did... [17:17:39] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338109 (10demon) >>! In T101358#1338106, @mmodell wrote: > If we stored the mapping in phabricator then a bit of custom code could look up the phabricator calls... [17:17:53] twentyafterfour: I was thinking of doing it as a Phab application. [17:18:14] Since we'd have all the data onhand other than the Gerrit bit, rather than a $separate_thing that would have to store Phab & Gerrit info [17:23:02] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338121 (10mmodell) @demon: so we could make a url mapping in phabricator that was essentially https://phabricator.wikimedia.org/r/old/gerrit/path and route th... [17:23:47] ostriches: yeah exactly [17:23:53] that's what I'm saying too ;) [17:27:12] http://fpaste.org/228962/14334388/raw/ <-- so many 302's [17:29:50] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338134 (10demon) >>! In T101358#1338071, @Ricordisamoa wrote: >>>! In T101358#1336865, @hashar wrote: >> What about having the Phabfile template to use the call... [17:30:49] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338140 (10demon) >>! In T101358#1338121, @mmodell wrote: > @demon: so we could make a url mapping in phabricator that was essentially https://phabricator.wikim... [17:33:10] legoktm: yeah I really don't like it and I complained at the time but we (the original phabricator team) were told that security through obscurity wasn't acceptable [17:34:00] twentyafterfour: what's the security issue? [17:35:17] files protected by policy - it has to validate the policy which requires passing a token across the domain boundary and validating it on the other side [17:35:38] so it double-redirects [17:35:54] it's a fast redirect at least but it is definitely a mess IMO [17:36:43] okay [17:36:49] well, python-requests handles it fine so :D [17:36:55] csteipp pointed out a few possible optimizations, I'm not sure if they would all work with phab but it's probably possible to reduce the burden of 302s to a lot less files [17:39:13] it is slightly madness but yeah it was the best worst solution :) [17:39:19] worst best? [17:46:56] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338204 (10Nemo_bis) > {{cn}} on not being usable or used. There was a wide consensus on http://markmail.org/message/jtdipertspvwdgpd [17:53:19] 10Browser-Tests, 6Mobile-Web: refactor "upload file" step to mediawiki_selenium gem - https://phabricator.wikimedia.org/T64888#1338224 (10Jdlrobson) Uploads code is gone from mobile, not sure if you guys want to add this to the selenium gem for future use. [17:53:43] 10Browser-Tests, 6Release-Engineering: refactor "upload file" step to mediawiki_selenium gem - https://phabricator.wikimedia.org/T64888#1338225 (10Jdlrobson) [17:53:57] 10Browser-Tests, 6Mobile-Web: Remove browser tests for file uploads - https://phabricator.wikimedia.org/T91992#1338227 (10Jdlrobson) 5Open>3Resolved a:3Jdlrobson This is done. [18:03:58] (03CR) 10Krinkle: [C: 032] Make SemanticForms unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh) [18:04:02] (03PS2) 10Krinkle: Make SemanticForms unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh) [18:04:08] (03CR) 10Krinkle: [C: 032] Make SemanticForms unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh) [18:05:45] (03Merged) 10jenkins-bot: Make SemanticForms unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh) [18:06:05] (03CR) 10Krinkle: "Fixed jshint regression in I48a44fda6d5924ff8." [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh) [18:06:20] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/214501 [18:06:24] Logged the message, Master [18:13:19] Yippee, build fixed! [18:13:19] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #671: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/671/ [18:24:16] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338343 (10demon) >>! In T101358#1338204, @Nemo_bis wrote: >> {{cn}} on not being usable or used. > > There was a wide consensus on http://markmail.org/message/... [19:01:32] marxarelli: I want to document this @skip thing, are you available? [19:01:56] spagewmf: yep! [19:02:09] in person or virtual? [19:03:13] marxarelli: I see you added extensions/Gather/tests/browser/features/support/hooks.rb , but I don't see where it's used. [19:05:50] spagewmf: oh, i'm wfh today. if you want to hangout, i available in about 15 min [19:13:28] marxarelli: you're invited at 12:30pm [19:19:10] spagewmf: rad! see ya htne [19:19:10] *then* [19:19:11] wow, double transposition [19:19:20] ya htne! xelcentee [19:19:24] haha [19:19:33] sorry, ahah [19:29:56] ahha [19:31:02] marxarelli: heading over [20:30:23] 6Release-Engineering, 5MW-1.25-release: No patch download for 1.25.1 - https://phabricator.wikimedia.org/T101404#1338753 (10Aklapper) [20:56:59] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1338885 (10mmodell) The alternative to callsigns in phabricator will be to use an integer. `R1`, `R2`, ... `R9001` see https://secure.phabricator.com/T4245#53345 [21:07:45] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<30.00%) [21:14:48] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #660: FAILURE in 48 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/660/ [21:23:12] legoktm: https://www.mediawiki.org/w/index.php?title=Special:Log&page=User%3AForrestbot [21:27:46] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [21:34:47] lol [21:34:49] oh! [21:34:54] I have the extensions code ready [21:34:57] I just need to deploy it [21:35:01] Do it! [21:35:02] ;-) [21:35:19] But maybe not do a manual run for all of wmf9 in one go? [21:35:25] That'd be quite a lot of activity. [21:36:16] it'll just start working for patches merged since it last ran at 2:00 [21:36:29] That's a start. [21:36:42] But it'd be nice to fill out some of the other ones. [21:38:56] we don't really have an easy way to back-fill [21:39:10] we could git log all the repositories, but that sounds painful [21:43:10] most evil option: git log, then mail the bot in a format that looks enough like a real gerrit change ;-) [21:43:11] legoktm: Just run for mail since 2015-05-28? [21:43:11] Oh, but we don't have those e-mails. Bah. [21:43:11] yeah, it pulls them with pop3 and DELEs them [21:43:42] https://phabricator.wikimedia.org/T97663#1339048 [21:43:57] Yay. [21:44:50] legoktm: I'm wondering whether we should just remove the comment (re. Nemo_bis' comment on getting spammed) [21:45:05] doesn't adding the project still spam? [21:45:12] legoktm: Yes. [21:45:18] yeah, but you can ignore 'add project' stuff in your settings [21:45:21] legoktm: But you can ignore… yes. [21:45:22] ignoring comments is less advisable [21:45:25] 10Deployment-Systems, 6Release-Engineering, 10ReleaseTaggerBot, 7Epic: EPIC: Code Deploy Dashboard - https://phabricator.wikimedia.org/T280#1339059 (10Legoktm) [21:47:00] I like the comments, but if other people don't we can get rid of it [21:47:24] I think they're especially useful to summarize what happened to backported patches [21:48:12] I like them too. [22:00:37] 10Browser-Tests, 6Release-Engineering: Provide support for spoffing the physical geolocation on QA tests - https://phabricator.wikimedia.org/T60720#1339134 (10Jdlrobson) [22:01:23] 10Browser-Tests, 6Collaboration-Team, 10Echo, 10Flow, 6Mobile-Web: Move user agent assignment to Ruby gem - https://phabricator.wikimedia.org/T73531#1339145 (10Jdlrobson) Any updates on this? @dduval that works for me. [22:14:57] twentyafterfour: Was upload-size-limit removed in master Phab? [22:15:07] storage., that is [22:17:57] Yep [22:24:22] bd808: Is *.local.wmftest.net broken? [22:24:25] Or is it just me? [22:25:01] dig foo.local.wmftest.net works for me [22:25:14] yeah, same here [22:25:18] (works for me) [22:26:11] foo.local.wmftest.net. 3599 IN A 127.0.0.1 [22:27:46] Ah nvm [22:27:50] All fine [22:40:36] 6Release-Engineering, 10Gather, 6Mobile-Web, 10MobileFrontend, and 2 others: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#1339352 (10Tgr) [22:40:57] 10Browser-Tests, 6Collaboration-Team, 10Echo, 10Flow, 6Mobile-Web: Move user agent assignment to Ruby gem - https://phabricator.wikimedia.org/T73531#1339353 (10dduvall) p:5Low>3High No updates, but let's bump the priority and move it up in TODO. [23:21:29] 10Browser-Tests, 10Gather, 6Mobile-Web, 10MobileFrontend, 3Mobile-Web-Sprint-48-Voyage-of-the-Damned: Audit existing browser tests - https://phabricator.wikimedia.org/T101071#1339512 (10Jdlrobson) 5Open>3Resolved Looks like all builds are green right now. Let's keep them that way. Calling this closed... [23:31:19] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:23] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL 100.00% of data above the critical threshold [43200.0] [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:53:33] (03PS1) 10Dduvall: Collapse steps when filtering by status [integration/raita] - 10https://gerrit.wikimedia.org/r/216010