[00:01:03] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1451044 (10BBlack) >>! In T102566#1450297, @demon wrote: >>>! In T102566#1449713, @BBlack wrote: >> So, where are we... [00:10:39] (03CR) 10BryanDavis: Don't assume current l10n cache files are .cdb (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 (owner: 10Ori.livneh) [00:11:35] (03CR) 10Ori.livneh: Don't assume current l10n cache files are .cdb (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 (owner: 10Ori.livneh) [00:28:37] (03PS4) 10Ori.livneh: Don't assume current l10n cache files are .cdb [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 [00:37:33] (03PS5) 10Ori.livneh: Don't assume current l10n cache files are .cdb [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 [00:39:01] (03CR) 10BryanDavis: [C: 032] Don't assume current l10n cache files are .cdb [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 (owner: 10Ori.livneh) [00:39:24] (03Merged) 10jenkins-bot: Don't assume current l10n cache files are .cdb [tools/scap] - 10https://gerrit.wikimedia.org/r/224520 (owner: 10Ori.livneh) [00:41:23] !log Updated scap to d7db8de (Don't assume current l10n cache files are .cdb) [00:41:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:41:54] !log trebuchet deploy of scap to mediawiki02 failed. investigating [00:41:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:45:29] !log /srv/deployment/scap/scap on deployment-mediawiki02 had corrupt git cache info; moved to scap-corrupt and forced a re-sync [00:45:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:20:14] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1367811 (10demon) >>! In T102566#1451044, @BBlack wrote: > Just to clear up confusion with the @Tgr's comment as wel... [01:58:38] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1451187 (10BBlack) Change prepped so it's easy. I'm open to debate on timing (I tend to think we should least have... [03:24:04] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #750: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/750/ [04:25:12] (03PS1) 10Ori.livneh: Don't exclude PHP files from being synced [tools/scap] - 10https://gerrit.wikimedia.org/r/224561 [04:25:40] (03CR) 10Ori.livneh: "Simple; line removed was added by previous patch." [tools/scap] - 10https://gerrit.wikimedia.org/r/224561 (owner: 10Ori.livneh) [04:25:46] (03CR) 10Ori.livneh: [C: 032] Don't exclude PHP files from being synced [tools/scap] - 10https://gerrit.wikimedia.org/r/224561 (owner: 10Ori.livneh) [04:26:06] (03Merged) 10jenkins-bot: Don't exclude PHP files from being synced [tools/scap] - 10https://gerrit.wikimedia.org/r/224561 (owner: 10Ori.livneh) [05:18:43] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1451278 (10Nemo_bis) > I tend to think we should least have a software release available The necessary steps also m... [06:39:18] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:43:05] 10Continuous-Integration-Infrastructure, 6Release-Engineering: automatically build and commit mediawiki/vendor (composer) - https://phabricator.wikimedia.org/T101123#1451314 (10JanZerebecki) I created an RFC (T105638) where this task might be one of the solutions. I'm open to other solutions to the problems ou... [06:54:35] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [06:59:03] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 100.00% of data above the critical threshold [0.0] [08:11:46] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [08:15:16] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace.root.byte_percentfree (<50.00%) [08:31:26] PROBLEM - SSH on deployment-bastion is CRITICAL - Socket timeout after 10 seconds [08:36:15] RECOVERY - SSH on deployment-bastion is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [08:36:43] Project beta-scap-eqiad build #61462: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61462/ [09:20:03] Project beta-update-databases-eqiad build #1458: FAILURE in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/1458/ [10:43:01] Project beta-code-update-eqiad build #63908: FAILURE in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63908/ [10:54:08] Yippee, build fixed! [10:54:09] Project beta-code-update-eqiad build #63909: FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63909/ [11:13:01] Project beta-code-update-eqiad build #63911: FAILURE in 0.11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63911/ [11:24:20] Yippee, build fixed! [11:24:20] Project beta-code-update-eqiad build #63912: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63912/ [12:05:29] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<40.00%) [12:07:17] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:54:43] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #533: FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/533/ [12:56:57] PROBLEM - Puppet failure on integration-slave-precise-1013 is CRITICAL 40.00% of data above the critical threshold [0.0] [12:59:17] PROBLEM - Puppet failure on integration-puppetmaster is CRITICAL 30.00% of data above the critical threshold [0.0] [12:59:55] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 50.00% of data above the critical threshold [0.0] [13:00:58] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 40.00% of data above the critical threshold [0.0] [13:02:54] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 30.00% of data above the critical threshold [0.0] [13:05:05] PROBLEM - Puppet failure on integration-publisher is CRITICAL 66.67% of data above the critical threshold [0.0] [13:05:38] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 60.00% of data above the critical threshold [0.0] [13:05:44] poor puppet. [13:09:44] PROBLEM - Puppet failure on deployment-salt is CRITICAL 90.00% of data above the critical threshold [0.0] [13:14:08] Project beta-code-update-eqiad build #63923: FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63923/ [13:16:45] And yet puppet runs fine on -fluorine [13:17:55] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [13:24:09] Yippee, build fixed! [13:24:10] Project beta-code-update-eqiad build #63924: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63924/ [13:34:52] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [13:36:56] RECOVERY - Puppet failure on integration-slave-precise-1013 is OK Less than 1.00% above the threshold [0.0] [13:39:21] RECOVERY - Puppet failure on integration-puppetmaster is OK Less than 1.00% above the threshold [0.0] [13:40:05] RECOVERY - Puppet failure on integration-publisher is OK Less than 1.00% above the threshold [0.0] [13:40:43] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [13:40:55] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [13:44:43] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [14:34:53] There is no change in SWAT/mw train during Wikimania, right? [14:35:20] kart__: Correct. [14:35:32] James_F: thanks! [14:36:18] scheduled something in next SWAT :) [14:41:53] * ostriches deploys all the things during Wikimania [14:41:55] bwahaha [14:43:01] Project beta-code-update-eqiad build #63932: FAILURE in 97 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63932/ [14:54:08] Yippee, build fixed! [14:54:08] Project beta-code-update-eqiad build #63933: FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63933/ [15:13:01] Project beta-code-update-eqiad build #63935: FAILURE in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63935/ [15:14:13] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:24:04] Yippee, build fixed! [15:24:04] Project beta-code-update-eqiad build #63936: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63936/ [15:40:29] ostriches: ori turned on a new kind of l10n cache in prod. If you get a chance, tack him down, beat him with a trout and get him to setup the same thing in beta cluster so we can actually test it :/ [15:40:38] *track [15:41:09] something something airplane? [15:41:24] yeah me too :) [15:41:46] but you know he will be back on irc 20m after he lands [15:42:35] My fear is that it is not wired up to build the cache on tin yet and we will get out of sync with the nightly updates (and maybe event the train) [15:42:54] I may be wrong about that, but then again I may not [15:43:01] Project beta-code-update-eqiad build #63938: FAILURE in 0.11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63938/ [15:46:49] * bd808 vanishes in a puff of airport wifi [15:50:57] (03PS1) 10Ori.livneh: Expect l10n_cache-en.php, not l10n_cache-en.cdb [tools/scap] - 10https://gerrit.wikimedia.org/r/224629 [15:54:03] Yippee, build fixed! [15:54:03] Project beta-code-update-eqiad build #63939: FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63939/ [15:54:15] (03CR) 10BryanDavis: [C: 031] "We need to make beta cluster use this" (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224629 (owner: 10Ori.livneh) [16:14:08] Project beta-code-update-eqiad build #63941: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63941/ [16:24:04] Yippee, build fixed! [16:24:04] Project beta-code-update-eqiad build #63942: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63942/ [16:37:44] (03PS1) 1020after4: check for dblist to validate arguments, [tools/release] - 10https://gerrit.wikimedia.org/r/224636 [17:19:19] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: Error "Duplicate declaration: File[/etc/ssh/userkeys] is already declared in file /private/modules/passwords/manifests/init.pp:36; cannot redeclare at /modules/ssh/manifests/server.pp:31" - https://phabricator.wikimedia.org/T92752#1452173... [17:25:16] PROBLEM - Puppet failure on nodepool-t105406 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:44:06] Project beta-code-update-eqiad build #63950: FAILURE in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63950/ [17:54:03] Yippee, build fixed! [17:54:04] Project beta-code-update-eqiad build #63951: FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63951/ [18:14:15] Project beta-code-update-eqiad build #63953: FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63953/ [18:24:10] Yippee, build fixed! [18:24:10] Project beta-code-update-eqiad build #63954: FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63954/ [18:25:22] (03PS1) 10Galorefitz: Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) [18:30:40] (03CR) 10Galorefitz: "Since the sniff also checks for empty single line comments, I am wondering if the file name should be something else. I couldn't come up w" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:36:00] (03CR) 10Polybuildr: "Thanks for the patch! :D" (033 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:36:12] (03CR) 10Polybuildr: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:36:31] (03CR) 10jenkins-bot: [V: 04-1] Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:38:54] (03CR) 10Polybuildr: "Right. Embarrassing when the tests fail because codesniffer doesn't pass codesniffer..." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:40:47] (03CR) 10Polybuildr: "The test run against core seems to have aborted due to some mysterious error involving composer not being able to download a file. Let's t" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:41:06] (03CR) 10Polybuildr: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:48:14] (03CR) 10Polybuildr: "Wow, Jenkins thought the error log was so long that it showed only the last part of the output. :/ But I looked through some of them - the" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [19:24:47] integration testing is failing for me on composer.lock errors: [19:24:49] https://integration.wikimedia.org/ci/job/mwext-CirrusSearch-testextension-zend/584/console [19:43:01] Project beta-code-update-eqiad build #63962: FAILURE in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63962/ [19:54:06] Yippee, build fixed! [19:54:06] Project beta-code-update-eqiad build #63963: FIXED in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63963/ [20:14:35] Project beta-code-update-eqiad build #63965: FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63965/ [20:15:30] what's up with scap on beta today? [20:16:15] 20:14:35 Caused by: java.io.IOException: remote file operation failed: /mnt/home/jenkins-deploy/workspace/beta-code-update-eqiad at hudson.remoting.Channel@34396ac9:deployment-bastion.eqiad: java.io.IOException: No space left on device [20:24:16] Yippee, build fixed! [20:24:17] Project beta-code-update-eqiad build #63966: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63966/ [20:44:00] Project beta-code-update-eqiad build #63968: FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63968/ [20:53:59] Yippee, build fixed! [20:53:59] Project beta-code-update-eqiad build #63969: FIXED in 57 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63969/ [21:13:01] Project beta-code-update-eqiad build #63971: FAILURE in 0.15 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/63971/ [21:24:37] Anyone releasing anything? [21:24:41] testwiki l10n cache seems borked [21:26:25] Also [21:26:42] Who decided minifying wikiversions.json as a good idea? [21:29:19] is it autogenerated file? [21:31:04] everyone knows that minimalism makes everything prettier [21:31:11] even when it gets completely unusable [21:37:38] maybe someone just forgot to add some flags as pretty print is not the default [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]