[00:17:53] ^d: could I have access to sudo as jenkins-slave as well? [00:18:15] <^d> You should? [00:18:19] <^d> I gave you +admin [00:19:50] https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration doesn't say I'm an admin [00:19:56] legoktm@integration-slave1005:~$ sudo -u jenkins-slave echo "hi" [00:19:56] [sudo] password for legoktm: [00:25:29] <^d> Special:NovaProject says you are [00:27:12] legoktm is not allowed to run sudo on integration-slave1005. This incident will be reported. [00:27:14] :/ [00:27:37] does integration have a custom sudo policy? [00:28:15] ^d: aha, https://wikitech.wikimedia.org/wiki/Special:NovaSudoer add me to "default"? [00:28:16] wouldn't be surprised that it does [00:28:54] <^d> done [00:29:46] woot, working now :D [00:29:49] thanks again :) [00:49:09] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [00:55:21] We have a cowbuilder in CI? [00:55:22] O_O [00:55:35] debian related apparently [00:55:42] crazy debian people [01:09:17] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<22.22%) [01:11:24] !log gzip'd /var/log/account/pacct.0 on deployment-bastion to free space [01:11:26] Logged the message, Master [01:24:10] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [01:25:44] looks like deployments reek some serious havoc on the processing accounting log [01:53:59] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1080743 (10dduvall) 3NEW [01:54:44] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1080759 (10dduvall) [02:14:55] Project beta-scap-eqiad build #43660: FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43660/ [02:34:55] Yippee, build fixed! [02:34:56] Project beta-scap-eqiad build #43662: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43662/ [03:18:32] (03PS1) 10Legoktm: Disambiguator extension needs VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/194036 [03:21:16] (03CR) 10Legoktm: [C: 032] Disambiguator extension needs VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/194036 (owner: 10Legoktm) [03:24:59] (03PS1) 10Krinkle: mw-install-sqlite: Expose tmpfs dir as TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) [03:28:05] (03Merged) 10jenkins-bot: Disambiguator extension needs VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/194036 (owner: 10Legoktm) [03:29:08] (03PS2) 10Krinkle: mw-install-sqlite: Expose tmpfs dir as TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) [03:29:29] (03CR) 10Krinkle: [C: 04-1] "Proposing Ife1585ed2f737234bdb7f6 instead." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/193832 (https://phabricator.wikimedia.org/T91070) (owner: 10Hashar) [03:31:47] (03CR) 10Krinkle: "Intended for deploy on March 3 during GMT afternoon." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) (owner: 10Krinkle) [03:32:10] Yippee, build fixed! [03:32:10] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #513: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/513/ [03:33:57] 10Continuous-Integration, 6Release-Engineering, 10MediaWiki-File-management, 6Multimedia: Parser tests intermittently failing on Zend due to unexpected thumbnail error - https://phabricator.wikimedia.org/T91016#1080939 (10Krinkle) Might be caused by T91070. [03:34:28] (03CR) 10Krinkle: [C: 032] "Might as well test now." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) (owner: 10Krinkle) [03:34:47] (03PS3) 10Krinkle: mw-install-sqlite: Expose tmpfs dir as TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) [03:34:56] (03CR) 10Krinkle: [C: 032] mw-install-sqlite: Expose tmpfs dir as TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) (owner: 10Krinkle) [03:35:28] (03Merged) 10jenkins-bot: mw-install-sqlite: Expose tmpfs dir as TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) (owner: 10Krinkle) [03:39:56] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [03:40:56] 10Continuous-Integration, 5Patch-For-Review: Jenkins: point TMPDIR to dir in workspace or tmpfs and delete after build - https://phabricator.wikimedia.org/T70563#1080946 (10Krinkle) 5Open>3Resolved a:3Krinkle [03:41:43] 10Continuous-Integration, 5Patch-For-Review: Jenkins: point TMPDIR to dir in workspace or tmpfs and delete after build - https://phabricator.wikimedia.org/T70563#727272 (10Krinkle) p:5Low>3Unbreak! [03:41:58] 10Continuous-Integration, 5Patch-For-Review: Jenkins: point TMPDIR to dir in workspace or tmpfs and delete after build - https://phabricator.wikimedia.org/T70563#727272 (10Krinkle) p:5Unbreak!>3Low [03:42:41] 10Continuous-Integration, 10MediaWiki-Unit-tests: MediaWiki Jobs running concurrently on the same instance share the same $wgTmpDirectory causing race condition - https://phabricator.wikimedia.org/T91070#1080954 (10Krinkle) 5Open>3Resolved [03:47:08] !log git-deploy: Deploying integration/slave-scripts 05a5593..1e64ed9 [03:47:12] Logged the message, Master [03:59:56] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.016 second response time [04:38:11] 6Release-Engineering, 10MediaWiki-Logging, 10Wikimedia-General-or-Unknown: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#1081023 (10Krinkle) [05:23:40] Yippee, build fixed! [05:23:41] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #220: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/220/ [05:29:11] (03Abandoned) 10Krinkle: Use jsonchecker.py for jsonlint macro [integration/config] - 10https://gerrit.wikimedia.org/r/192060 (https://phabricator.wikimedia.org/T73284) (owner: 10Legoktm) [05:29:46] Krinkle: did you see https://phabricator.wikimedia.org/T73284#1058027 ? [05:30:22] legoktm: It applies to phplint as well. [05:30:55] why? [05:31:27] legoktm: Because we're not going to run a complete job just to run a few lint checjs. [05:31:32] It's just part of the new entry point. [05:31:40] And also makes it a lot easier to run locally. [05:32:13] and yes, that means all -in-all we will need to consolidate and migrate lots of jobs, including all extensions. [05:32:14] So the alternative is to configure 600+ repositories individually just to check php -l * ? [05:32:22] Once the new jobs are stable and tested for a few weeks, we'll replace them all with a composer -composer job. [05:33:00] It doesn't matter that it's a lot of repos. [05:33:18] Is running "a complete job" that expensive? It's only a few seconds [05:33:27] Not if you need to use a VM. [05:33:43] right, but we're not using vm's yet. [05:33:48] And it's still overhead and a complete mw clone in some cases depending on where the last job ran and if it's concurrent. [05:33:53] And again, easier to run locally. [05:34:09] Consistency. [05:34:34] Not really, if my system has PHP 5.6 installed, I'd have to build my own version of PHP to lint against 5.3. [05:34:52] Sure. [05:35:04] But it's conceptually the same kind of tests at least. [05:35:15] It will also run on Mac instead of Linux if you run it there. [05:35:34] And having json included is definitely desireable. [05:35:52] And already common practice for every other project on earth that uses travis. And also for standalone projects here using -npm. [05:36:03] No exceptions. [05:36:38] dunno, last time I checked VE wasn't linting JSON [05:37:24] It does, [05:37:36] I think it would be useful to have an individual job, until we move to VMs at least. I'm already looking into implementing it in PHP... [05:38:05] The only potentially dormant json files in VE that warrant linting (e.g. not package.json which is run by npm install), are i18n files. Which are linted and checked by banana [05:38:48] legoktm: I'd rather the job stays as-is. Any early bird project can migrate to -npm or -composer and add a jsonlint check there. And we'll need to incorporate it there somehow. [05:39:17] mediawiki-core also uses -npm already and validates json with banana via grunt [05:39:40] For example an older patch of https://gerrit.wikimedia.org/r/#/c/189864/ introducted a syntax error in extension.json, which was caught by PHPUnit, not a linter. [05:40:05] Krinkle: and for non-whitelisted users? [05:40:57] legoktm: non-whitelisted users were not a high enough priority to WMF. That's why labs forged ahead with open registrations, and we locked down CI until we have resources to do VM testing. [05:41:14] Just doesn't matter. [05:41:31] Skilled new contributors should be able to run npm/composer test locally anyway. [05:42:24] I don't agree with that all [05:42:48] If we're trying to recruit newcomers (which we are), we should be lowering the barrier to entry, not raising it [05:43:01] I'm not saying we shouldn't run tests for newcomers. [05:43:14] Hence we finally, after 2 years of me pushing personally, we're working on isolation. [05:43:31] However trying to make it work without it is crazy, unscalable and unreasonable. [05:44:00] Upgrading jshint globally was a bitch, all 6 times. [05:44:31] and also why phpcs is practically non-voting for many projects because we kept changing it. [05:44:46] Proper CI has no state and runs tests as declared by the local repo. [05:45:18] Unisolated jenkins installs are for corporate environments were all users are authorised. [05:45:41] I understand why we need isolation, but until we have VMs I think we still need something for non-whitelisted users, which is basically syntax checks for now [05:47:28] I think we've gone in a full circle that I'm no longer sure what I'm arguing about. [05:48:20] You want to replace jsonlint job with a python implementation with an additional feature to warn for duplicate keys. [05:48:42] I think we need a check for duplicated json keys, and don't see isolated VMs happening fast enough to make it worth postponing even further. [05:49:06] Isolated VMs or not isn't directly related. [05:49:19] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #500: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/500/ [05:49:44] the jsonlint macro and job are not used by mediawiki-core and many other projects. Only for non-whitelist users as fallback in certain cases. [05:50:03] It's used in most MW extensions [05:50:03] So it wouldn't even improve the test effectively for the main pipeline [05:50:23] Yes. The majority of unmaintained extensions with little to no tests. [05:50:38] active job tests that is. I'm sure they have unit testst. [05:52:24] legoktm: I'm fine with replacing the job implementation for now. But it won't resolve the bug effectively. We'll still need a different approach for the early-bird mediawiki projects such as core, editing repos, as well as non-mediawiki repos such as cdb etc. [05:52:34] unrelated to isolated vms. [05:53:03] thus needing the same behaviour in either a grunt or composer test. [05:53:26] I'm working on a PHP version that would be runnable via composer [05:55:10] https://github.com/Seldaek/jsonlint supports checking for duplicate keys, it just doesn't expose it in the CLI utility it comes with, so I'm working on a patch for that [05:55:24] cool [05:55:40] we wouldn't even have to migrate to composer to use that [05:56:01] like we do for some legacy node modules, we can install the composer package in integration/jenkins.git:/tools [05:56:08] and run it from the current entry point [05:56:51] that would make it consistent and for nice migration [05:57:23] I'm interested in the performance difference...the python one is mostly C and a little bit of Python whereas this is pure PHP [05:57:48] Yeah, but how is its error reporting though? [05:58:03] The reason the js lib jsonlint when with a lexer is to have detailed parse errors [05:58:12] I know the default one in PHP and JS don't have that otherwise [05:58:21] What about the Python one? [05:59:22] it just uses whatever error message the standard library provides, I gave an example https://phabricator.wikimedia.org/T73284#938359. It's usually useful for me to figure out where the error is, but idk how it compares to jsonlint [05:59:33] k [06:08:26] (03PS1) 10Krinkle: jsduck: Remove redundant overrides for oojs-core and oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/194052 [06:10:35] (03PS1) 10Krinkle: jsduck: Move '{name}-jsduck' from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194053 (https://phabricator.wikimedia.org/T86174) [06:13:13] (03PS2) 10Krinkle: jsduck: Move '{name}-jsduck' from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194053 (https://phabricator.wikimedia.org/T86174) [06:16:45] (03PS3) 10Krinkle: jsduck: Move '-jsduck' jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194053 (https://phabricator.wikimedia.org/T86174) [06:16:47] (03PS1) 10Krinkle: jsduck: Move '-jsduck-publish' jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194054 (https://phabricator.wikimedia.org/T86174) [06:21:18] (03CR) 10Krinkle: [C: 032] "No-op." [integration/config] - 10https://gerrit.wikimedia.org/r/194052 (owner: 10Krinkle) [06:27:32] (03CR) 10Abartov: "So, my patch included upping the version number, but one of [Dan, Zeljko, Chris] still needs to publish the new gem to rubyforge for clien" [ruby/api] - 10https://gerrit.wikimedia.org/r/183200 (owner: 10Abartov) [06:27:59] (03Merged) 10jenkins-bot: jsduck: Remove redundant overrides for oojs-core and oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/194052 (owner: 10Krinkle) [06:29:14] (03PS1) 10Krinkle: jsduck: Remove override for 'VisualEditor-jsduck' [integration/config] - 10https://gerrit.wikimedia.org/r/194056 [06:34:30] (03CR) 10Krinkle: [C: 032] "Deployed oojs-core-jsduck, oojs-ui-jsduck, unicodejs-jsduck, and VisualEditor-jsduck." [integration/config] - 10https://gerrit.wikimedia.org/r/194053 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [06:41:06] (03Merged) 10jenkins-bot: jsduck: Move '-jsduck' jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194053 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [06:44:18] (03PS1) 10Krinkle: jsduck: Remove obsolete --process=0 override for Ruby 1.8 [integration/config] - 10https://gerrit.wikimedia.org/r/194058 (https://phabricator.wikimedia.org/T62138) [07:03:27] (03CR) 10Krinkle: [C: 04-1] "Depends on I51674dee." [integration/config] - 10https://gerrit.wikimedia.org/r/194058 (https://phabricator.wikimedia.org/T62138) (owner: 10Krinkle) [07:10:39] (03PS1) 10Krinkle: jsduck: Move 'mwext-GuidedTour-doc-test' from gallium to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194061 (https://phabricator.wikimedia.org/T86174) [07:17:09] (03CR) 10Krinkle: [C: 032] "Deployed mwext-GuidedTour-doc-test." [integration/config] - 10https://gerrit.wikimedia.org/r/194061 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [07:20:34] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release4, 3LE-Sprint-83: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1081232 (10KartikMistry) @greg @hashar What the best we can do here? [07:23:50] (03Merged) 10jenkins-bot: jsduck: Move 'mwext-GuidedTour-doc-test' from gallium to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194061 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [07:25:52] (03PS2) 10Krinkle: jsduck: Move '-jsduck-publish' jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194054 (https://phabricator.wikimedia.org/T86174) [07:28:20] (03PS3) 10Krinkle: jsduck: Move jsduck-publish jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194054 (https://phabricator.wikimedia.org/T86174) [07:29:09] (03PS2) 10Krinkle: jsduck: Remove override for 'VisualEditor-jsduck' [integration/config] - 10https://gerrit.wikimedia.org/r/194056 [07:32:53] Krinkle|detached: jsonchecker (python) checks languages/i18n/*.json in .7s, jsonlint (php) takes 2m13s :/ [07:36:10] hhvm brought it down to 1m31s [07:42:56] (03PS1) 10Krinkle: jsduck: Remove jsduck-conf override for mwext-GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/194065 [07:48:49] (03PS1) 10Krinkle: jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 [07:56:33] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:01:19] (03CR) 10Adrian Lang: "Possibly, although in this case none of the dependencies are MediaWiki extensions in need of update.php execution. `prepare-mediawiki` als" [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [08:01:49] (03CR) 10Krinkle: [C: 032] "Deployed oojs-ui-jsduck-publish, oojs-core-jsduck-publish, and mwext-GuidedTour-doc-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/194054 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [08:06:04] (03PS3) 10Krinkle: jsduck: Remove jsduck-conf override for VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/194056 [08:08:14] (03Merged) 10jenkins-bot: jsduck: Move jsduck-publish jobs from gallium to labs/trusty slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194054 (https://phabricator.wikimedia.org/T86174) (owner: 10Krinkle) [08:08:50] 10Continuous-Integration, 5Patch-For-Review: Fetch dependencies using composer instead of cloning mediawiki/vendor repository for non-WMF deployment branches - https://phabricator.wikimedia.org/T90303#1081259 (10adrianheine) In addition to phpunit, we would need a similar job for qunit, too. [08:11:03] (03PS2) 10Krinkle: jsduck: Remove jsduck-conf override for mwext-GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/194065 [08:11:21] (03PS2) 10Krinkle: jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 [08:15:06] (03CR) 10Krinkle: [C: 032] "Deployed VisualEditor-jsduck." [integration/config] - 10https://gerrit.wikimedia.org/r/194056 (owner: 10Krinkle) [08:17:49] (03CR) 10Krinkle: [C: 032] "Deployed mwext-GuidedTour-doc-test and mwext-GuidedTour-doc-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/194065 (owner: 10Krinkle) [08:21:54] (03Merged) 10jenkins-bot: jsduck: Remove jsduck-conf override for VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/194056 (owner: 10Krinkle) [08:24:31] (03Merged) 10jenkins-bot: jsduck: Remove jsduck-conf override for mwext-GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/194065 (owner: 10Krinkle) [08:30:11] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-03, 10§ Wikidata-Sprint-2015-02-25: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1081282 (10adrianheine) AFAIK we are the only ones with focus tests. [08:34:44] Project beta-scap-eqiad build #43697: FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43697/ [08:39:15] (03CR) 10Zfilipin: [C: 032] Setup Gather browser tests job [integration/config] - 10https://gerrit.wikimedia.org/r/193393 (https://phabricator.wikimedia.org/T91082) (owner: 10Jdlrobson) [08:39:23] (03PS4) 10Zfilipin: Setup Gather browser tests job [integration/config] - 10https://gerrit.wikimedia.org/r/193393 (https://phabricator.wikimedia.org/T91082) (owner: 10Jdlrobson) [08:39:31] (03CR) 10Zfilipin: [C: 032] Setup Gather browser tests job [integration/config] - 10https://gerrit.wikimedia.org/r/193393 (https://phabricator.wikimedia.org/T91082) (owner: 10Jdlrobson) [08:39:46] (03CR) 10Zfilipin: "And the job is green! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/193393 (https://phabricator.wikimedia.org/T91082) (owner: 10Jdlrobson) [08:43:30] 10Continuous-Integration, 10MediaWiki-ResourceLoader, 10MediaWiki-Vagrant, 10Wikidata, and 2 others: qunit test broken without explicitly setting $wgResourceLoaderMaxQueryLength - https://phabricator.wikimedia.org/T90453#1081310 (10adrianheine) >>! In T90453#1073327, @Krinkle wrote: > * The relevant softwa... [08:46:07] (03Merged) 10jenkins-bot: Setup Gather browser tests job [integration/config] - 10https://gerrit.wikimedia.org/r/193393 (https://phabricator.wikimedia.org/T91082) (owner: 10Jdlrobson) [08:54:58] Yippee, build fixed! [08:54:58] Project beta-scap-eqiad build #43699: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43699/ [08:55:04] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-03, 10§ Wikidata-Sprint-2015-02-25: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1081318 (10adrianheine) The failing ExpertExtender tests also need focus apparently. [08:55:52] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25: Make ValueView qunit tests independent of focus. - https://phabricator.wikimedia.org/T91376#1081320 (10adrianheine) 3NEW [08:56:30] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25: Make Wikibase qunit tests independent of focus - https://phabricator.wikimedia.org/T91377#1081327 (10adrianheine) 3NEW [08:57:39] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25: Make Wikibase qunit tests independent of focus - https://phabricator.wikimedia.org/T91377#1081327 (10adrianheine) [08:57:40] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-03, 10§ Wikidata-Sprint-2015-02-25: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1081334 (10adrianheine) [09:01:27] hello [09:23:39] 10Continuous-Integration: post-merge build fails with doxygen - https://phabricator.wikimedia.org/T91382#1081390 (10Amire80) 3NEW [09:37:16] 10Beta-Cluster: Parser cache (memcached?) broken in Beta Cluster - https://phabricator.wikimedia.org/T91310#1081426 (10hashar) The memcached instances haven't been rebuild. The memcached and nutcracker configurations seems fine to me. I have no idea what is going on :( [09:40:13] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: Jenkins qunit builds fail for many ContentTranslation patches and complain about EventLogging - https://phabricator.wikimedia.org/T91383#1081435 (10Amire80) 3NEW [09:40:51] we'd need some help debugging that ^^ [09:41:11] I'm unable to reproduce locally [09:42:53] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1081455 (10hashar) [09:42:54] 10Beta-Cluster, 10Wikimedia-Labs-Infrastructure, 7Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#1081454 (10hashar) [09:44:51] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: Jenkins qunit builds fail for many ContentTranslation patches and complain about EventLogging - https://phabricator.wikimedia.org/T91383#1081457 (10Nikerabbit) There are many things in the log and I am not sure which things are relevant and w... [09:45:24] hashar: https://phabricator.wikimedia.org/T91383 , as Nikerabbit already said... it blocks all our development. [09:45:28] thanks in advance :) [09:53:18] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1081469 (10hashar) We lowered the accounting retention with T71604 . That is done by having puppet populate `/etc/default/acct` with `ACCT_LOGGING="7"`... [09:55:15] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release4, 3LE-Sprint-83: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1081476 (10hashar) @KartikMistry I guess you have a valid use case t... [10:02:07] (03PS1) 10Hashar: mwcore-docgen: fix invalid whitespace [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194085 [10:03:12] (03CR) 10Hashar: [C: 032] mwcore-docgen: fix invalid whitespace [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194085 (owner: 10Hashar) [10:03:40] (03Merged) 10jenkins-bot: mwcore-docgen: fix invalid whitespace [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194085 (owner: 10Hashar) [10:14:13] 10Continuous-Integration: post-merge build fails with doxygen - https://phabricator.wikimedia.org/T91382#1081517 (10hashar) From the console output, the job ran on integration-slave1007 (phpflavor-hhvm contintLabsSlave UbuntuTrusty) ``` [mediawiki-core-doxygen-publish] $ /bin/bash -xe /tmp/hudson55502691090987... [10:14:59] (03CR) 10Hashar: "That seems to have broken the mediawiki-core-doxygen-publish job. wfTempDir() ends up returning false or an empty string: T91382" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194038 (https://phabricator.wikimedia.org/T70563) (owner: 10Krinkle) [10:22:10] 10Continuous-Integration: post-merge build fails with doxygen - https://phabricator.wikimedia.org/T91382#1081529 (10hashar) On integration-slave1007: ``` $ sudo su - jenkins-deploy $ cd /mnt/jenkins-workspace/workspace/mediawiki-core-doxygen-publish $ export WORKSPACE= `pwd` $ . /srv/deployment/integration/slav... [10:25:50] 10Continuous-Integration: post-merge build fails with doxygen - https://phabricator.wikimedia.org/T91382#1081537 (10hashar) p:5Triage>3High [10:26:13] 10Continuous-Integration: mediawiki/extensions.git does not update some extensions - https://phabricator.wikimedia.org/T51846#1081541 (10hashar) p:5Unbreak!>3Normal [10:26:16] 10Continuous-Integration: Jenkins merged a faulty change - https://phabricator.wikimedia.org/T48723#1081544 (10hashar) p:5Unbreak!>3Normal [10:26:19] 10Continuous-Integration: Jenkins merged a faulty change - https://phabricator.wikimedia.org/T48723#513584 (10hashar) p:5Normal>3Unbreak! [10:26:22] 10Continuous-Integration: mediawiki/extensions.git does not update some extensions - https://phabricator.wikimedia.org/T51846#586052 (10hashar) p:5Normal>3Unbreak! [10:41:56] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: Jenkins qunit builds fail for many ContentTranslation patches and complain about EventLogging - https://phabricator.wikimedia.org/T91383#1081569 (10hashar) The MediaWiki logs are capture for each builds and attached to the build result. https... [10:45:00] aharoni: Nikerabbit: hello! I replied on https://phabricator.wikimedia.org/T91383 [10:45:10] seems there is some breaking change in mediawiki/core [10:52:46] !log deleting integration-puppetmaster to recreate it with a new image {bug|T87484} . Will have to reapply I5335ea7cbfba33e84b3ddc6e3dd83a7232b8acfd and I30e5bfeac398e0f88e538c75554439fe82fcc1cf [10:52:52] Logged the message, Master [10:56:06] !log Created instance i-000008fb with image "ubuntu-14.04-trusty" and hostname i-000008fb.eqiad.wmflabs. [10:56:08] Logged the message, Master [10:57:53] 10Continuous-Integration, 5Patch-For-Review: Recreate integration-puppetmaster with new image (/var/ is too small) - https://phabricator.wikimedia.org/T87484#1081580 (10hashar) I have deleted integration-puppet master . Will have to reapply on operations/puppet the changes I5335ea7cbfba33e84b3ddc6e3dd83a7232b8... [11:26:29] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-03, 10§ Wikidata-Sprint-2015-02-25: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1081607 (10adrianheine) [11:26:30] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25: Make Wikibase qunit tests independent of focus - https://phabricator.wikimedia.org/T91377#1081604 (10adrianheine) 5Open>3Invalid a:3adrianheine They already are, it's just ValueView. [11:27:31] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25, 5Patch-For-Review: Make ValueView qunit tests independent of focus. - https://phabricator.wikimedia.org/T91376#1081608 (10adrianheine) https://github.com/wmde/ValueView/pull/161 [11:28:17] 10Continuous-Integration, 10Wikidata, 10§ Wikidata-Sprint-2015-02-25, 5Patch-For-Review: Make ValueView qunit tests independent of focus. - https://phabricator.wikimedia.org/T91376#1081616 (10adrianheine) p:5Normal>3High [11:32:37] 10Beta-Cluster: Caching makes it impossible to test JS changes when logged out - https://phabricator.wikimedia.org/T65034#1081622 (10yuvipanda) /me pokes resident varnish expert @bblack [11:54:46] Yippee, build fixed! [11:54:46] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #174: FIXED in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/174/ [12:16:19] (03PS1) 10Zfilipin: Bumped patch version to 0.3.1 [ruby/api] - 10https://gerrit.wikimedia.org/r/194097 (https://phabricator.wikimedia.org/T91388) [12:24:15] (03CR) 10Zfilipin: [C: 032] "Self merging because the change is trivial and I am releasing the gem." [ruby/api] - 10https://gerrit.wikimedia.org/r/194097 (https://phabricator.wikimedia.org/T91388) (owner: 10Zfilipin) [12:24:38] (03Merged) 10jenkins-bot: Bumped patch version to 0.3.1 [ruby/api] - 10https://gerrit.wikimedia.org/r/194097 (https://phabricator.wikimedia.org/T91388) (owner: 10Zfilipin) [12:37:36] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [13:40:20] Krinkle: hello! The integration project has a new puppetmaster with a large / partition [13:40:35] Yup. Just saw it. [13:40:37] instance has the same name [13:40:38] Cool [13:45:14] (03PS1) 10Zfilipin: Update Chrome, Firefox and Safari to the latest supported version [integration/config] - 10https://gerrit.wikimedia.org/r/194101 (https://phabricator.wikimedia.org/T91389) [13:45:23] cowbuilder --execute --basepath /packaging/base-trusty.cow/ /bin/cat /etc/lsb-release|grep DISTRIB_RELEASE [13:45:23] DISTRIB_RELEASE=14.04 [13:46:28] (03PS1) 10Krinkle: Implement mw-setup.sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194102 (https://phabricator.wikimedia.org/T91382) [13:48:38] (03PS1) 10Krinkle: Add mw-setup and mw-teardown to mediawiki-core-doxygen-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) [13:50:29] (03CR) 10jenkins-bot: [V: 04-1] Add mw-setup and mw-teardown to mediawiki-core-doxygen-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [13:51:53] (03PS2) 10Krinkle: Add mw-setup and mw-teardown to mediawiki-core-doxygen-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) [13:55:15] (03PS2) 10Zfilipin: Update Chrome, Firefox and Safari to the latest supported version [integration/config] - 10https://gerrit.wikimedia.org/r/194101 (https://phabricator.wikimedia.org/T91389) [14:02:27] hashar: around? [14:02:48] is jenkins-jobs update really slow lately? or is it just me? [14:02:57] it takes about a minute to update just one job [14:05:26] zeljkof: that's just Jenkins. [14:05:50] Krinkle: is it slow today? or in general? [14:05:50] API is lowest priority in its web server. This because it has to write stuff to disk and can't be done while running any related job. [14:05:57] Always been that way. [14:06:00] Krinkle: I see [14:06:02] Speed varies on load, very much. [14:06:10] I do not remember it being so slow in the past [14:06:21] maybe it is just busy right now [14:06:22] at the right time it's very fast [14:06:34] If it takes more than 30 seconds I usually abort and run again [14:08:02] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #384: FAILURE in 7 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/384/ [14:25:37] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #31: FAILURE in 9 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/31/ [14:30:41] (03CR) 10Zfilipin: "I have tested it with a few jobs, looks like everything is fine:" [integration/config] - 10https://gerrit.wikimedia.org/r/194101 (https://phabricator.wikimedia.org/T91389) (owner: 10Zfilipin) [14:32:41] Yippee, build fixed! [14:32:41] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #386: FIXED in 7 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/386/ [14:47:44] (03PS1) 10Hashar: Logrotate tox jobs after 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) [14:47:57] hashar: https://gerrit.wikimedia.org/r/#/c/194102/ and https://gerrit.wikimedia.org/r/#/c/194103/ [14:48:00] good to go? [14:50:48] (03CR) 10Hashar: "Maybe it should be in the defaults directly? What do you think Timo?" [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [14:51:38] Krinkle: bah more context switch :) [14:51:42] (03CR) 10Krinkle: "Yeah, let's remove it from everything that has >=30 days and use that as default." [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [14:53:18] (03CR) 10Hashar: [C: 031] "That is clever and I love it. Deploy it at anytime :)" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194102 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [14:53:33] (03CR) 10Krinkle: [C: 032] Implement mw-setup.sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194102 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [14:54:07] (03Merged) 10jenkins-bot: Implement mw-setup.sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/194102 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [14:55:36] 00:00:00.117 + '[' -z ']' [14:55:36] 00:00:00.117 + echo '$DOC_SUBPATH is missing. Can not publish.' [14:55:37] 00:00:00.117 $DOC_SUBPATH is missing. Can not publish. [14:55:40] hashar: still failing. [14:55:43] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/8983/console [14:55:49] (03CR) 10Krinkle: [C: 032] "Deployed mediawiki-core-doxygen-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [14:55:58] (03CR) 10Hashar: [C: 031] "Good to me. No need to update the mw-install-sqlite.sh calls since it has been updated to invoke mw-setup.sh in https://gerrit.wikimedia.o" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [14:56:11] hashar: I'll look into it. Could you check the zuul-cloner intsall failure? [14:56:16] Krinkle: ah yeah [14:56:24] Krinkle: DOC_SUBPATH is provided by Zuul directly [14:56:42] if it ends up being empty that might have some weird side effect when the content is rsynced [14:57:43] Sre [14:57:43] Krinkle: you want to hit Rebuild on the previous build https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/8981/parameters/ [14:57:44] Sure [14:57:48] I know [14:57:50] That's what I did [14:58:29] ahhh [14:58:31] stupid Jenkins [14:58:50] Rebuild doesnt get the whole list of parameters that actually got used to trigger the job [14:58:51] bah [14:59:05] OK [14:59:08] We'll see after next comit [14:59:13] let's not worry [14:59:26] or you can manually enqueue a change in the postmerge pipeline [15:00:01] on gallium: zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/core --change 192599,2 [15:00:23] that will hook into Zuul scheduler and craft an event in postmerge [15:00:38] nice [15:01:01] its working now [15:02:27] (03Merged) 10jenkins-bot: Add mw-setup and mw-teardown to mediawiki-core-doxygen-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194103 (https://phabricator.wikimedia.org/T91382) (owner: 10Krinkle) [15:03:26] congratulations! [15:10:49] Yippee, build fixed! [15:10:49] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #34: FIXED in 9 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/34/ [15:12:01] (03PS2) 10Hashar: Logrotate jobs after 30 days by default [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) [15:16:01] (03PS1) 10Hashar: Lower mediawiki-gate logrotation from 90 to 7 days [integration/config] - 10https://gerrit.wikimedia.org/r/194115 (https://phabricator.wikimedia.org/T91396) [15:18:18] (03PS1) 10Hashar: Drop logrotation from 90 days to 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194116 (https://phabricator.wikimedia.org/T91396) [15:19:41] (03CR) 10Hashar: "This now being applied at the default level." [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [15:25:34] Krinkle: do you think https://gerrit.wikimedia.org/r/#/c/180418/ can be merged now? (so reworking the ci to always run composer on master, etc. can be done in new patches) [15:31:26] (03PS2) 10Hashar: Drop logrotation from 90 days to 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194116 (https://phabricator.wikimedia.org/T91396) [15:31:28] (03PS2) 10Hashar: Lower mediawiki-gate logrotation from 90 to 7 days [integration/config] - 10https://gerrit.wikimedia.org/r/194115 (https://phabricator.wikimedia.org/T91396) [15:31:30] (03PS3) 10Hashar: Logrotate jobs after 30 days by default [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) [15:31:43] !log updating all jobs in Jenkins based on PS2 of https://gerrit.wikimedia.org/r/194109 [15:31:45] Logged the message, Master [15:33:39] (03CR) 10Hashar: [C: 032] "Being deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [15:34:45] (03CR) 10Hashar: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/194115 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [15:39:02] !log cancelled logrotate update of all jobs since that seems to kill the Jenkins/Zuul gearman connection. Probably because all jobs are registered on each config change. [15:39:04] Logged the message, Master [15:42:59] jzerebecki: I'm not sure. Currently fixing three other outages unfortunately. [15:45:40] (03Merged) 10jenkins-bot: Logrotate jobs after 30 days by default [integration/config] - 10https://gerrit.wikimedia.org/r/194109 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [15:45:41] (03Merged) 10jenkins-bot: Lower mediawiki-gate logrotation from 90 to 7 days [integration/config] - 10https://gerrit.wikimedia.org/r/194115 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [15:45:58] (03CR) 10Krinkle: [C: 031] Drop logrotation from 90 days to 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194116 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [16:02:39] well [16:02:46] off for some groceries shopping [16:02:53] will be back in an hour for the weekly checkin [16:08:11] hashar: Did you leave? zuul looks stuck... [16:10:24] Sigh [16:12:32] (03CR) 10Krinkle: "Depends on mediawiki-core-jsduck and a few others migrating first as well." [integration/config] - 10https://gerrit.wikimedia.org/r/194058 (https://phabricator.wikimedia.org/T62138) (owner: 10Krinkle) [16:12:53] Did someone restart zuul? [16:12:58] It's not stuck, it's down [16:13:02] like literally [16:13:20] Why would you restart zuul [16:14:28] There are a dozen known issues that can cause a stuck queue. None of them warrant restarting of zuul though. That purges the queue also. And likely still leaves gearman stuck [16:14:45] https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Known_issues [16:15:26] I don't see anything in the log [16:15:29] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:15:39] I see in -operations now [16:16:21] Sorry. [16:16:49] ah [16:16:52] Krinkle: Zuul looked stuck to me - I'll add a note on the wiki about that [16:17:07] marktraceur: Zuul is a light frontend. "It" is rarely stuck. It's just blocked by something else. [16:17:36] there is a note about not restarting zuul: [16:17:37] "Do not restart Zuul when deploying a configuration change. Zuul can reread configuration from disk while running using the reload command (See Reload). This way no Gerrit events are missed. Do not take restarting Zuul lightly, as it means any Gerrit events during that time will be missed and need to be manually re-submitted to trigger tests (and merging)." [16:17:37] restarting it drops the live queue, and also creates a 1-2 minutes were incoming events will be missed. [16:17:49] No worries though, there wasn't much going on [16:17:52] :) [16:17:57] I'm not aware of another way to fix it, but OK [16:18:00] early morning in SF, no one's awake yet [16:18:06] read down to known issues? [16:18:06] Hopefully we can add another way to my note. [16:18:22] *sigh* gj mark [16:18:32] :) [16:18:35] s'ok, [16:18:36] g'morning [16:18:40] marktraceur: I can't speak for past documentation, but to my knowledge restarting zuul has never been a fix. However told you that was wrong. [16:19:13] So please remove it from your note if that helps :) [16:19:51] Krinkle: Well, it did "fix" my issue, and I do remember doing that a while ago, maybe 6-12 months [16:20:09] What was the issue exactly? [16:20:38] It might've just taken 1-2 minutes longer, and worked now because there was no waiting queue after you're first in line post-restart. [16:20:44] I don't see any sign that it was stuck. [16:20:50] Anyway, we'll see. [16:22:35] We're currently under capacity while we're re-creating the slave pool. This should've taken only a few minutes last Thursday, but due to our puppet manifests getting out of sync with upstream, it wasn't trivial to create new instances. There's two remaining issues with zuul-cloner install. One I fixed yesterday, the other hashar is looking into. [16:23:02] greg-g: ^ (tracked as T90984) [16:23:31] where's wikibugs [16:23:41] Re-creating instances is a good way of ensuring puppet is working properly. [16:23:43] :D [16:23:52] :) [16:24:26] that's going to be our motto for the staging cluster: recreate recreate recreate [16:25:35] Krinkle: There was a log entry about Gearman not being connected, that was probably the issue, the config change definitely took too long to merge, it was stalled at "SUCCESS" for several minutes [16:26:07] Restarting the Gearman connection (via Jenkins UI, two clicks) should resolve that. [16:31:38] Right. [16:32:03] And now I know. And even better, when I forget, and go back to the zuul page, there will be a big yellow box telling me how stupid I am. [16:32:34] Because I think you told me this once before, and I forgot, and didn't add any documentation about how stupid I was. [16:34:35] greg-g: "Recreate! Let no instances evade the gate! Just recreate recreate recreate… Only be sure always to call it, please, 'clean-up'!". [16:36:08] I'm not getting the reference, if there is one [16:36:40] greg-g: http://www.sing365.com/music/lyric.nsf/Lobachevsky-lyrics-Tom-Lehrer/D97B21BF6516390448256A7D0024B8B9 [16:37:00] greg-g: Tom Lehrer. Should be mandatory stuff. [16:38:59] (03PS1) 10Krinkle: Remove mediawiki-phpunit-zend from mediawiki-core 'test' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194134 [16:40:40] (03PS3) 10Krinkle: jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 [16:41:06] (03CR) 10Jforrester: [C: 031] Remove mediawiki-phpunit-zend from mediawiki-core 'test' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194134 (owner: 10Krinkle) [16:41:37] greg-g: I modified https://www.mediawiki.org/w/index.php?title=Wikimedia_Release_Engineering_Team/Process#Teams_and_Tools to point out that Editing also does do SWATs. :-) [16:41:41] (Hope that's OK.) [16:45:38] James_F: 'Once the instances are up, who cares about their clean-up? That's not my department,' says Puppet [16:45:42] sure, don't take that page as gospel [16:46:00] * jzerebecki even has a tshirt about that [16:46:29] jzerebecki: Nice. :-) [16:56:44] I write puppet. Puppet writes me. [16:57:12] Works so well in theory. [17:01:04] twentyafterfour: ^d meeting ping : [17:01:06] er :) [17:01:10] <^d> Joining [17:02:54] (03CR) 10Krinkle: [C: 032] "Deployed mwext-VisualEditor-jsduck, mwext-VisualEditor-jsduck-publish, mwext-MultimediaViewer-jsduck, mwext-MultimediaViewer-jsduck-publis" [integration/config] - 10https://gerrit.wikimedia.org/r/194066 (owner: 10Krinkle) [17:04:16] joining sorry [17:09:48] (03CR) 10jenkins-bot: [V: 04-1] jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 (owner: 10Krinkle) [17:11:29] (03CR) 10Krinkle: [C: 032] jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 (owner: 10Krinkle) [17:18:07] (03Merged) 10jenkins-bot: jsduck: Rename remaining doc-test/doc-publish to jsduck/jsduck-publish [integration/config] - 10https://gerrit.wikimedia.org/r/194066 (owner: 10Krinkle) [17:18:20] (03PS2) 10Krinkle: Remove mediawiki-phpunit-zend from mediawiki-core 'test' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194134 [17:18:25] (03CR) 10Krinkle: [C: 032] Remove mediawiki-phpunit-zend from mediawiki-core 'test' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194134 (owner: 10Krinkle) [17:18:41] !log Reloading Zuul to deploy Icad0a26dc8 and Icac172b16 [17:18:43] Logged the message, Master [17:19:28] (03Merged) 10jenkins-bot: Remove mediawiki-phpunit-zend from mediawiki-core 'test' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194134 (owner: 10Krinkle) [17:22:12] (03Abandoned) 10Hashar: Remove phpunit-zend from mediawiki-core test pipeline (kept in gate) [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar) [17:22:26] (03CR) 10Hashar: "Awesome!" [integration/config] - 10https://gerrit.wikimedia.org/r/194134 (owner: 10Krinkle) [17:24:57] <^d> thcipriani: Finished the security groups for staging [17:25:31] sweet. just saw the email about increasing groups. [17:43:53] bah [17:43:58] KDE Kmix confuses me [17:44:10] the webRTC record stream was on mute [17:44:56] and I learned about acpi_listen command ! [17:45:06] jack/headphone HEADPHONE plug [17:45:06] jack/microphone MICROPHONE plug [17:45:06] looks nice [17:46:41] oh, you'll learn to {love,hate} all those acpi events with linux desktop stuff :) https://github.com/thcipriani/acpi [17:50:05] (03CR) 10Krinkle: "Fixes T86175." [integration/config] - 10https://gerrit.wikimedia.org/r/194066 (owner: 10Krinkle) [17:52:10] hashar: Anything I can help with? Currently migrating some more jsduck jobs. [17:52:14] zuul-cloner :) [17:54:35] thcipriani: nice [17:54:41] Krinkle: going to package Zuul this week [17:55:01] I have work envs locally to build packages for Precise and Trusty :D [17:55:07] gotta play with /debian/ now :=( [17:55:21] hashar: Hm.. Is there not something we can do to make the puppet manifest work? [17:55:21] will loop ops with it [17:55:31] nop [17:55:35] It's installed on the old instances. [17:55:36] how did that work> [17:55:38] What changed. [17:55:45] we need a bunch of dependencies to be shipped which are not packaged or not in apt.wikimedia.org [17:56:03] at least one dep (python-six) require a more recent version than what we have in apt.wm.o :( [17:56:19] hashar: Does it work on Precise, but just not in apt? We can install with pip, right? [17:56:21] It's a labs instance. [17:56:22] I did a terrible hack [17:56:29] shipping the dependencies as tarball [17:56:34] directly in integration/zuul.git [17:57:06] Krinkle: gotta prepare dinner sorry. In short: keep moving stuff out of gallium/lanthanum :) [17:57:35] hashar: Hm.. I want to dismantle the old instances today or tomorrow. [17:57:46] Doesn't need a Debian package. And we already worked around it on the old instances. [17:57:47] Krinkle: if the new ones works fine. Sure go ahead [17:58:01] hasharDinner: So on the new precise instances, I can install it with pip? [17:58:03] I'm not sure how. [17:58:12] lets catch up later tonight [17:58:16] I guess run setup without proxy? [17:58:17] OK :) [17:58:19] should be there around 9pm (hopefully) [17:58:28] ++ [17:58:30] Bon Appetit :) [17:58:42] merci! [18:02:20] hi! are there any known problems with the browser tests and firefox 36? [18:05:18] I'm going to go to a drs appt then to the DMV (for previously mentioned RV), be back... later [18:20:00] thcipriani: looks like your ops puppet patches were merged. any other SoS blockers you need addressed? [18:20:17] (03PS1) 10Krinkle: jsduck: Move mediawiki-core-jsduck from gallium to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194153 (https://phabricator.wikimedia.org/T86175) [18:20:22] greg-g: oh boy. good luck! [18:20:41] marxarelli: nope, andrewbogott got 'em after the meeting. [18:20:49] greg-g: don't let the dmv bully you [18:20:53] thcipriani: nice! [18:28:33] !log Lots of Jenkins builds are stuck half-way executing. No clear cause. Everything seems up. [18:28:35] Logged the message, Master [18:30:30] !log The stuck builds have a line of "Finished: .." in them, but are still showing a loader spinner on the bottom of their build log 10 minutes later. [18:30:32] Logged the message, Master [18:40:06] (03PS3) 10Krinkle: Remove mediawiki-core-regression-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193583 (https://phabricator.wikimedia.org/T88018) [18:41:43] (03PS4) 10Krinkle: Remove mediawiki-core-regression-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193583 (https://phabricator.wikimedia.org/T88018) [18:42:18] (03CR) 10Krinkle: [C: 032] Remove mediawiki-core-regression-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193583 (https://phabricator.wikimedia.org/T88018) (owner: 10Krinkle) [18:48:45] !log New builds run fine, but there's 30 stuck builds occupying executors. Their output is finished and Zuul/Gerrit got the event already, but they won't die. [18:48:47] Logged the message, Master [18:48:51] wtf, never seen this [18:56:18] (03Merged) 10jenkins-bot: Remove mediawiki-core-regression-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193583 (https://phabricator.wikimedia.org/T88018) (owner: 10Krinkle) [19:03:02] (03PS2) 10Krinkle: jsduck: Move mediawiki-core-jsduck from gallium to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194153 (https://phabricator.wikimedia.org/T86175) [19:06:20] (03PS3) 10Krinkle: jsduck: Move mediawiki-core-jsduck from gallium to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194153 (https://phabricator.wikimedia.org/T86175) [19:07:27] ^d: Are all the gerrit repos in diffusion now? I can't seem to find mediawiki/vagrant with the search [19:07:33] <^d> Not yet [19:07:37] <^d> I need to finish that off now [19:07:45] ah. ok [19:09:24] <^d> Actually, lemme do them now [19:09:25] <^d> I have a script [19:13:29] ^d: Woo, even VisualEditor? :-) [19:14:41] <^d> Yes [19:16:46] <^d> James_F: https://phabricator.wikimedia.org/diffusion/GVED/ [19:17:18] <^d> "A tool for editing visually!" [19:17:22] <^d> I hope you keep it :) [19:18:55] <^d> bd808: Ask and ye shall receive https://phabricator.wikimedia.org/diffusion/MWVA/ [19:19:17] sweet. Thanks ^d [19:19:22] <^d> yw [19:19:58] awwww... it doesn't pretty render .md files -- https://phabricator.wikimedia.org/diffusion/MWVA/browse/master/support/README-lxc.md [19:20:20] <^d> Oh, it does for the main readme in the repo root [19:20:28] <^d> I guess others just get highlighting [19:21:08] meh. still a better place to link than to github in the email I'm writing [19:35:39] <^d> James_F: https://phabricator.wikimedia.org/diffusion/query/.ZBIyadoJB9M/ [19:39:15] <^d> Also, if you could re-run https://www.mediawiki.org/wiki/Phabricator/Diffusion/Callsign_naming_conventions/Existing_repositories with a list of repos in Gerrit not in Phab, that'd be nice. I could keep updating that rows but there's probably new repos too since we made the table last [19:40:49] ooh, stuff's getting imported. [19:40:57] so much lovely email! <3 [19:41:00] <^d> Almost everything's imported [19:44:00] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:45:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:54:15] (03CR) 10Krinkle: [C: 032] "Deployed mediawiki-core-jsduck." [integration/config] - 10https://gerrit.wikimedia.org/r/194153 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [19:54:19] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:00:42] (03Merged) 10jenkins-bot: jsduck: Move mediawiki-core-jsduck from gallium to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/194153 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [20:04:06] (03PS1) 10Krinkle: Move mwext-MultimediaViewer-jsduck and mwext-VisualEditor-jsduck to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194174 (https://phabricator.wikimedia.org/T86175) [20:08:53] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #4: FAILURE in 2 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/4/ [20:18:50] ^d: Woohoo. [20:19:11] ^d: "Re-run" means "re-build manually", you understand. :-) [20:19:20] <^d> :) [20:33:15] !log manually created centralauth.users_to_rename table [20:33:19] Logged the message, Master [20:39:40] Yippee, build fixed! [20:39:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #501: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/501/ [20:40:27] ^d: OK, I give up. [20:40:37] ^d: OpenOffice has now hard-crashed five times trying to do this. [20:40:42] ^d: Atom three. [20:40:47] ^d: Nano once. [20:40:52] ^d: You do it. :-P [20:41:04] * ^d goes to eat lunch instead [20:41:09] have you tried rebooting? [20:41:40] legoktm: It turns out trying to parse ~5MB of HTML being pasted in isn't something any of these do well. [20:41:48] :P [20:46:00] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/194174 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [20:52:18] (03Merged) 10jenkins-bot: Move mwext-MultimediaViewer-jsduck and mwext-VisualEditor-jsduck to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194174 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [21:00:58] ^d: fyi I don't think the jenkins tarballs include the skins yet...I don't think that code was ever merged. [21:04:59] Project beta-scap-eqiad build #43769: FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43769/ [21:11:43] Yippee, build fixed! [21:11:43] Project beta-scap-eqiad build #43770: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43770/ [21:25:27] James_F: try vim [21:25:54] hasharDinner: It is… poor at being an HTML interpreter. [21:26:22] James_F: but you can learn it via a game http://vim-adventures.com/ [21:26:36] you can even play golf http://www.vimgolf.com/ [21:26:37] :D [21:27:43] James_F: worse than nano? [21:27:59] preeeeetty sure people edit html with vim :) [21:28:17] Krinkle: I am quite happy to see mediawiki-phpunit-zend to be gone from the test pipeline [21:28:19] greg-g: Nano was a frustrated try, not a serious tool. [21:28:39] (03PS1) 10Krinkle: Move mwext-VisualEditor-jsduck-publish to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194193 (https://phabricator.wikimedia.org/T86175) [21:29:54] (03CR) 10Krinkle: "Even .git was being copied :D https://doc.wikimedia.org/VisualEditor/master/lib/ve/.git" [integration/config] - 10https://gerrit.wikimedia.org/r/194193 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [21:30:28] hashar: hey [21:31:15] James_F: here for you http://fte.sourceforge.net/shots.html [21:31:24] James_F: very similar to MS DOS good old editor [21:31:24] (03PS2) 10Krinkle: Move mwext-VisualEditor-jsduck-publish to labs integration-publisher [integration/config] - 10https://gerrit.wikimedia.org/r/194193 (https://phabricator.wikimedia.org/T86175) [21:31:47] hashar: TBNT. [21:35:43] (03CR) 10Krinkle: [C: 032] "Deployed mwext-VisualEditor-jsduck-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/194193 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [21:36:58] (03PS3) 10Hashar: Drop logrotation from 90 days to 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194116 (https://phabricator.wikimedia.org/T91396) [21:37:39] (03CR) 10Hashar: [C: 031] Update Chrome, Firefox and Safari to the latest supported version [integration/config] - 10https://gerrit.wikimedia.org/r/194101 (https://phabricator.wikimedia.org/T91389) (owner: 10Zfilipin) [21:39:04] (03CR) 10Jforrester: Update Chrome, Firefox and Safari to the latest supported version (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/194101 (https://phabricator.wikimedia.org/T91389) (owner: 10Zfilipin) [21:40:11] (03CR) 10Jforrester: [C: 031] Drop logrotation from 90 days to 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/194116 (https://phabricator.wikimedia.org/T91396) (owner: 10Hashar) [21:42:27] (03Merged) 10jenkins-bot: Move mwext-VisualEditor-jsduck-publish to labs integration-publisher [integration/config] - 10https://gerrit.wikimedia.org/r/194193 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [21:45:32] (03PS1) 10Krinkle: Move mwext-MultimediaViewer-jsduck-publish to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194198 (https://phabricator.wikimedia.org/T86175) [22:00:38] !log reboot integration-puppetmaster in case it solves a NFS mount issue [22:00:42] Logged the message, Master [22:01:26] ahh [22:01:26] ops [22:01:58] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: invalid byte sequence in US-ASCII at /etc/puppet/manifests/role/zotero.pp:1 on node i-000008fb.eqiad.wmflabs [22:03:44] it has unicode? [22:03:52] and newer puppet version has a problem with that [22:04:07] alex told me about this once in another context [22:04:40] like planet templates that have unicode [22:05:28] mutante: yeah it has a bom [22:05:33] mutante: filled as https://phabricator.wikimedia.org/T91453 :D [22:07:01] ok, thanks, i'll take a look. that also reminded me to check on that planet issue [22:07:27] (03PS6) 10Jforrester: Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 [22:07:29] (03CR) 10Krinkle: [C: 032] "Deployed mwext-MultimediaViewer-jsduck-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/194198 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [22:13:52] (03Merged) 10jenkins-bot: Move mwext-MultimediaViewer-jsduck-publish to labs [integration/config] - 10https://gerrit.wikimedia.org/r/194198 (https://phabricator.wikimedia.org/T86175) (owner: 10Krinkle) [22:16:47] interesting, it also has "illegal input sequence" for iconv [22:21:27] mutante: ohhh [22:22:04] hashar: try again :) [22:22:16] just merged a fix [22:22:24] mutante: I was looking for a BOM :D [22:22:45] i added those bad characters :p [22:22:52] by copy/pasting from ...Wikipedia :) [22:23:14] good I have Could not find class role::labs::instance now [22:23:18] but that is a different issue [22:23:23] hmm [22:23:29] ould not parse for environment production: invalid byte sequence in US-ASCII at /etc/puppet/manifests/role/planet.pp:1 [22:24:06] hashar: hah, wow [22:24:18] I am going to bed [22:24:19] so that is what i meant earlier, alex saw this [22:24:28] if you wanna test, it is on integration-puppetmaster.eqiad.wmflabs [22:24:36] which is a puppet master self, Ubuntu Trusty instance [22:24:36] and the difference must be upgrade of puppet version [22:24:40] might be [22:24:41] hashar: That error is caused by space [22:24:49] r: Error 400 on SERVER: [22:24:51] Usually, anyway. [22:25:03] nvm, that seems unlikely [22:25:08] i will try to fix the planet thing [22:25:28] it's cause it's utf-8 in there [22:25:28] not sure why it does not happen on the production puppetmaster though [22:25:35] but older pupppet didn't mind this [22:25:41] it's used in prod without that error [22:25:51] i may replace literal UTF8 with HTML entities [22:25:53] for this one [22:26:06] i just need them to generate HTML anyways [22:26:15] hashar: only because puppet version, pretty sure [22:30:20] hashar: Let me know about zuul cloner before you leave. [22:32:04] Krinkle: too late sorry. In short the deps aare provided as tarball in the source repo [22:32:09] and pip install should find them [22:32:17] cause the requirements.txt file point to the .tar.gz files [22:32:27] or fill in a task for me to look at tomorrow morning [22:32:35] as I understand it that is the last blocker [22:32:37] hashar: So I guess we upgraded to newer version of the code with additioanl dependencies not yet tarballed? [22:32:41] then I get to work on packaging zuul [22:32:46] hashar: I did that yesterday :P [22:32:50] ah [22:32:59] still have 200 phabricator emails to go [22:33:00] :( [22:33:15] https://phabricator.wikimedia.org/T90984 [22:33:24] "/usr/local/bin/zuul-cloner" broken on new instances [22:33:26] got it [22:33:36] hashar: It's unable to find statsd [22:33:57] hashar: My work around was going to be to exec pip install manaully on the server [22:34:05] but without the disabled http proxy, so it just fetches them [22:34:30] That'll be my workaround if it's not working tomorrow. My primary objective is to roll over to the new instances before everything goes stale again. [22:34:41] sure thing [22:34:42] ack [22:34:45] approve [22:34:48] +2 :D [22:35:02] I mean, it's a labs instance. a one-time setup [22:35:04] will fix tomorrow morning after I took care of breakfast / kids / coffee etc [22:35:09] Cool [22:35:13] if all fail [22:35:29] I will do some evil trick :) [22:36:29] well [22:36:34] going to try to get my computer to sleep [22:36:40] hopefully it will boot up tomorrow [22:36:45] * hashar waves [22:38:17] 10Continuous-Integration, 6Collaboration-Team, 10Flow: Flow: Off-by-one timestamp failures in tests - https://phabricator.wikimedia.org/T75833#1084708 (10Krinkle) [22:38:57] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1084710 (10Dzahn) [22:47:25] Krinkle: feel like running puppet on integration-puppetmaster? [22:47:31] i believe the error should be gone [22:47:37] mutante: Which error sorry? [22:47:57] invalid byte sequence in US-ASCII [22:48:17] this one https://phabricator.wikimedia.org/T91453#1084654 [22:48:31] mutante: Was that a regression from earlier today? there was no error yesterday. [22:48:55] Krinkle: i believe the puppet version must have changed recently [22:49:07] so, yes [22:49:14] version of puppet itself? [22:49:15] interesting [22:49:20] yes [22:49:27] the code did not change and is used like that in production [22:49:47] mutante: By running puppet I assume you mean rebasing our fork? [22:49:47] and Alex told me about this happening in a newer version [22:50:14] eh, yes, syncing it with the production puppetmaster [22:50:16] Hm.. how does puppet upgrade work in general btw? Does that happen from labs or something? [22:50:17] where i just merged a fix [22:50:56] i don't know yet how it happens in labs, but i imagine it has "ensure => latest" somewhere [22:51:05] in production it doesn't [22:51:25] mutante: seems it was already applied [22:51:31] No errors as of 5 hours ago [22:51:33] ah, or.. this is because we are using new instances in labs [22:51:34] by hashar [22:51:49] Presumably cherry-picked [22:52:05] eh, but this https://gerrit.wikimedia.org/r/#/c/194214/ [22:52:10] I'm not doing a full rebase right now as thatwill bring in lots of other patches I"m not prepared to deal with right now [22:52:12] i just merged that a minute after hashar left [22:52:31] mutante: It was applied to integration-puppetmaster 30 minutes ago [22:52:41] https://tools.wmflabs.org/nagf/?project=integration#h_overview_puppetagent [22:52:43] it did not exist 30 minutes ago [22:52:51] Uploaded2015-03-03 14:36 [22:52:57] I don't know what to tell you [22:53:09] There's no puppet failures in CI at the moment and the patch in question is there [22:53:21] I'll leave it as-is for now. Thanks for merging :) [22:54:06] 14:23 <+hashar> ould not parse for environment production: invalid byte sequence in US-ASCII at /etc/puppet/manifests/role/planet.pp:1 [22:54:18] it's not even 30 minutes ago that hashar reported the error [22:54:20] but ok [22:54:38] mutante: Ah, found it [22:54:46] Puppet is terrible at actually failing [22:54:57] it always annoyingly forges ahead and then not exit with error code [22:55:07] I see it on the logs now [22:55:14] as recent as 3 minutes ago it failed [22:55:15] yeah [22:55:22] Mar 3 22:54:28 integration-puppetmaster puppet-master[1325]: Could not parse for environment production: invalid byte sequence in US-ASCII at /etc/puppet/manifests/role/planet.pp:1 on node i-000005ce.eqiad.wmflabs [22:55:28] yes, that's the one [22:55:45] if we can https://gerrit.wikimedia.org/r/#/c/194214/ on it [22:55:51] that should make it go away [22:55:53] But this patch is already on it: [22:55:54] * e637dfe - Merge "zotero: fix "invalid byte sequence in US-ASCII"" into production (34 minutes ago) [22:55:59] that's not the same one [22:56:03] Ah, right [22:56:07] i made 2 fixes [22:56:16] hashar only saw the next error after the first was fixed [22:56:28] 6Release-Engineering, 15Team-Practices-This-Week: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1084756 (10Awjrichards) [22:56:44] mutante: yeah, sorry. got it. Rebased, and applied. [22:56:49] both are because of UTF-8 chars though [22:56:57] cool, thanks [22:57:01] Thank you. [22:57:47] let me know if the error changes (hopefully there are not more of the same kind) [22:57:51] 10Staging, 6operations: Package trebuchet-trigger for trusty - https://phabricator.wikimedia.org/T91463#1084831 (10thcipriani) 3NEW [22:58:55] 10Staging: Setup staging-tin as deployment host - https://phabricator.wikimedia.org/T88442#1084854 (10thcipriani) [22:58:56] 10Staging, 6operations: Package trebuchet-trigger for trusty - https://phabricator.wikimedia.org/T91463#1084855 (10thcipriani) [22:59:09] 6Release-Engineering, 6Team-Practices: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1084856 (10Awjrichards) [22:59:28] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1084857 (10Dzahn) a:3Dzahn [23:01:05] http://projects.puppetlabs.com/issues/20522 [23:01:44] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1084905 (10Dzahn) see http://projects.puppetlabs.com/issues/20522 and all the subtasks [23:02:38] 10Continuous-Integration, 5Patch-For-Review: Migrate jsduck jobs to run in labs (and jsduck-publish via integration-publisher) - https://phabricator.wikimedia.org/T86175#1084931 (10Krinkle) 5Open>3Resolved [23:02:39] 10Continuous-Integration: Migrate all jobs depending on Zuul git repos out of production slaves - https://phabricator.wikimedia.org/T86659#1084932 (10Krinkle) [23:02:46] 10Continuous-Integration: Migrate jsduck jobs to run in labs (and jsduck-publish via integration-publisher) - https://phabricator.wikimedia.org/T86175#962885 (10Krinkle) [23:04:52] 10Continuous-Integration: Migrate all jobs depending on Zuul git repos out of production slaves - https://phabricator.wikimedia.org/T86659#1084958 (10Krinkle) [23:05:31] 10Continuous-Integration: Migrate all jobs depending on Zuul git repos out of production slaves - https://phabricator.wikimedia.org/T86659#973540 (10Krinkle) [23:06:38] 10Continuous-Integration: Migrate all jobs to labs slaves - https://phabricator.wikimedia.org/T86659#1084960 (10Krinkle) [23:09:16] 10Continuous-Integration, 7Upstream: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1084964 (10Krinkle) p:5High>3Low [23:09:42] 10Continuous-Integration, 7Upstream: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#975087 (10Krinkle) Lowering priority. It doesn't happen very often and will be obsolete if we use clean workspaces and/or isolated VMs. [23:13:17] 10Continuous-Integration, 6Collaboration-Team, 10Flow: Flow: Off-by-one timestamp failures in tests - https://phabricator.wikimedia.org/T75833#1084969 (10Mattflaschen) Per our IRC conversation: For the testConverionToTimestamp, I think we can just do: UUID::create( 'somehex' )->getTimestamp() which wil... [23:28:57] ugh, right, https://phabricator.wikimedia.org/maniphest/query/86Ps9nQd9cg1/#R <-- that sorts by date created (newest -> oldest, can't change that apparently), but, it only goes back to the date of import to phab, which for queries with a lot of old bugs.... isn't helpful [23:29:02] (#wikimedia-log-errors) [23:44:25] <^d> greg-g: Speaking of log-errors: https://phabricator.wikimedia.org/T91466 [23:44:33] <^d> That one popped up today and is pretty freaking noisy [23:48:34] is it worth having brad take a look at that one, ^d ? [23:48:47] <^d> If he's got the cycles [23:57:56] how long do temp logstash search urls last? [23:58:13] eg https://logstash.wikimedia.org/#dashboard/temp/tZU_q1JzR9GNrCYXswUvgA