[00:02:44] !log deployment-puppetmaster:/var/lib/git/operations/puppet was left mid-rebase conflicts with portals apache config. Fixed conflicts added the file and continued the rebase. [00:02:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:13:55] 10Deployment-Systems, 3Scap3, 5Patch-For-Review, 7Puppet: Refactor `mediawiki::scap` to make sure Scap dependencies are not dependent on mediawiki - https://phabricator.wikimedia.org/T116606#1798095 (10thcipriani) Patch above is some work that I had ~80% complete before this task moved to in-progress. @joe... [00:14:53] 10Deployment-Systems, 3Scap3: Scap3 repo-cache should clean unused revs - https://phabricator.wikimedia.org/T118305#1798100 (10mmodell) yeah the workdir would still be a problem [00:20:07] Project beta-update-databases-eqiad build #4294: 04FAILURE in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4294/ [00:32:20] greg-g: do you know why http://en.wikipedia.beta.wmflabs.org/wiki/ displays Error Our servers are currently experiencing a technical problem. ? [00:33:17] Probably the same reason the database update failed [00:33:35] Reedy: yeah, probably [00:34:55] Reedy: this one? Could not connect to 10.64.53.12:9092 (Connection timed out [110]) [00:35:35] Error: 429, Request Rate Exceeded at Wed, 11 Nov 2015 00:35:27 GMT [00:35:38] Helpful [00:37:23] Reedy: ok - sigh [00:37:29] why are things falling apart today? :( [00:42:26] Do I dare start digging? [00:45:40] Reedy: you know you wanna :P [00:45:49] if not, thcipriani ostriches ^ [00:45:52] I've gotta be up in less than 8 hours :P [00:45:57] hah, well then [00:46:17] "better fix it quickly" [00:46:32] Hmm [00:46:52] greg-g: It's just gonna be that kind of week, I've already accepted it [00:46:56] * greg-g nods [00:47:15] oojs/oojs-ui: 0.13.2 installed, 0.13.1 required. [00:47:15] Error: your composer.lock file is not up to date. Run "composer update" to install newer dependencies [00:47:21] Blame James_F [00:47:39] yeah, carrie's appt today went longer than I planned (which meant I had to be on point for Rowan for longer) and then I got back to an absurd email from the workers comp claims adjuster (TMI, too long) [00:47:43] I wonder if it's because the core patch wasn't merged before vendor [00:48:17] * Reedy manually runs beta-scap-eqiad [00:48:24] yay composer [00:48:35] Wait, that won't update the code will it? [00:49:07] `sudo -u jenkins-deploy composer update --no-dev`? [00:49:50] Nah [00:49:57] The vendor repo is updated on deployment-bastion [00:49:59] core isn't [00:50:31] https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is what I want [00:51:03] Umm [00:51:06] What's broken? [00:51:16] root@deployment-bastion:/srv/mediawiki-staging/php-master# git pull [00:51:16] Updating cfdf523..d310101 [00:51:16] error: Your local changes to the following files would be overwritten by merge: [00:51:16] includes/resourceloader/ResourceLoader.php [00:51:16] Please, commit your changes or stash them before you can merge. [00:51:17] Aborting [00:51:30] Local changes to RL.php? [00:51:33] Whut? [00:51:34] Indeed [00:51:39] I don't think that's your fault :P [00:51:44] Imma blow away whoevers changes those are [00:51:44] No. [00:51:48] But it might be my problem. [00:51:50] Stash them. [00:51:55] Just in case. [00:53:20] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 - https://phabricator.wikimedia.org/T118158#1798164 (10Dzahn) a:3Dzahn [00:55:09] That should be it... [00:55:45] greg-g: https://etherpad.wikimedia.org/p/unsophabricated FYI. [00:56:00] Yippee, build fixed! [00:56:00] Project beta-code-update-eqiad build #80289: 09FIXED in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/80289/ [00:57:45] James_F: word. doit. +1. etc. [00:57:52] When beta-scap-eqiad finishes, that should fix it [00:57:57] greg-g: Woo. April/May for 1.27.0 sounds sane? [00:58:18] yeah [00:58:21] Kk. [00:59:44] (more like may, I think) [01:01:24] KK [01:01:31] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 - https://phabricator.wikimedia.org/T118158#1798168 (10Dzahn) wget http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.625.1_all.deb .. [carbon:/srv/wik... [01:01:57] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 - https://phabricator.wikimedia.org/T118158#1798169 (10Dzahn) 5Open>3Resolved [01:01:58] 10Continuous-Integration-Infrastructure, 7Jenkins, 7WorkType-Maintenance: Upgrade Jenkins to LTS 1.625.1 - https://phabricator.wikimedia.org/T118157#1798170 (10Dzahn) [01:05:03] How long does beta l10ncache update take [01:05:41] Hmm, how long is a piece of string? [01:05:43] How much changed? [01:05:51] Reedy: 45 minutes. [01:05:59] Reedy: Ish. Sometimes less, rarely more. [01:08:19] Varnish is making beta unseable anyway [01:08:20] :D [01:09:51] greg-g: You don't pay me enough [01:09:52] xD [01:17:58] beta puppetmaster automatically pulls doesn't it [01:18:00] ? [01:21:00] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #437: 04FAILURE in 3 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/437/ [01:23:26] Yippee, build fixed! [01:23:26] Project beta-update-databases-eqiad build #4296: 09FIXED in 3 min 25 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4296/ [01:25:13] Reedy: it does, it gets stuck with cherry picks sometimes. [01:30:42] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798211 (10Reedy) 3NEW [01:31:52] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798218 (10Reedy) p:5Triage>3High [01:32:03] Is this actually unbreak now [01:32:19] 00:56:15 00:56:15 Updating LocalisationCache for master using 2 thread(s) [01:32:19] 01:31:36 01:31:36 Generating JSON versions and md5 files [01:32:22] Finally [01:32:26] Not that anyone can use beta... [01:39:18] 01:38:51 01:38:51 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list. [01:39:35] guess that's normal [01:41:33] yeah, there are no proxies for beta. [01:41:47] 01:41:00 sync-common: 14% (ok: 0; fail: 1; left: 6) [01:41:58] 01:41:00 Finished: ABORTED [01:43:45] yeah, not sure why that is happening. That seems like it could only happen on the jenkins side, right? [01:43:58] seemingly, yeah [01:44:19] run it again, it should be a load quicker without the l10nupdates to do [01:45:47] I guess it's more broken due to varnish atm anyway [01:47:55] rebuilding cdbs [01:50:32] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798233 (10GWicke) Other victims are RESTBase integration tests, from either the office or travis. About 40 API requests distributed across a ~50 second test run. [01:50:50] success [01:50:51] Ok, done [01:51:05] Just the varnish error [01:51:26] Which... I really don't know much about [01:51:40] That, and getting up in 6.5 hours or so... I think I'm done for the night :) [01:51:50] Reedy: kk, thanks for your help! [01:52:34] np :) [02:02:12] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798242 (10Dzahn) Maybe oauth from labs is affected. Getting reports that login screens like: https://tools.wmflabs.org/wikidata-game/ using the "widar" tool... [02:03:28] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798243 (10GWicke) In case it helps with debugging, there seems to have been a multi-hour delay between the deploy & blocking starting in the office. [02:13:27] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798250 (10Dzahn) This works again http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page after Faidon restarted varnish in beta. [02:56:02] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1798315 (10GWicke) 5Open>3Resolved a:3GWicke Confirmed resolved for RESTBase tests as well. Most of those are actually hitting labs... [02:59:00] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #877: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/877/ [05:29:40] Yippee, build fixed! [05:29:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #599: 09FIXED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/599/ [08:34:03] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #779: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/779/ [09:57:26] PROBLEM - Host deployment-cache-parsoid04 is DOWN: CRITICAL - Host Unreachable (10.68.19.197) [10:06:54] 10Gitblit-Deprecate, 10Diffusion: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#1798692 (10mmodell) @demon: Do you have a list of the repository callsigns mapping to the old project urls? I can build the redirect script but we need to have that mapping somewhere... [10:12:27] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 - https://phabricator.wikimedia.org/T118158#1798710 (10hashar) Danke! ``` $ ssh gallium.wikimedia.org apt-cache policy jenkins jenkins: Installed: 1.625.1... [10:13:27] 10Continuous-Integration-Infrastructure, 7Jenkins, 7WorkType-Maintenance: Upgrade Jenkins to LTS 1.625.1 - https://phabricator.wikimedia.org/T118157#1798715 (10hashar) 5stalled>3Resolved a:3hashar @Dzahn uploaded the new version to apt.wikimedia.org: ``` $ ssh gallium.wikimedia.org apt-cache policy jen... [10:16:20] Hello zeljkof ! [10:16:32] hi vikassy! [10:17:44] zeljkof: I was thinking of adding few tasks related to Language screenshot(of covering the VisualEditor UserGuide 100%) to Google code in [10:17:48] what do you think ? [10:17:59] vikassy: go for it! :) [10:18:23] zeljkof: awesome ! :) [11:04:59] 7Browser-Tests, 10MediaWiki-extensions-Translate, 5Patch-For-Review, 7Ruby, 5WMF-deploy-2015-11-10_(1.27.0-wmf.6): Fix easy problems reported by RuboCop in Translate - https://phabricator.wikimedia.org/T117987#1798758 (10zeljkofilipin) 5Open>3Resolved [11:28:50] zeljkof: yay [11:28:56] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 3Wikidata-Sprint-2015-11-03: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1798808 (10Lucie) [11:29:47] Nikerabbit: what's up? [11:30:24] oh, commenting on the T117987? [11:38:46] zeljkof: yeah [12:41:01] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #845: 04FAILURE in 9 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/845/ [14:31:58] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #320: 04FAILURE in 3 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/320/ [14:32:37] 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-November-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1799021 (10Qgil) p:5Normal>3High [16:05:17] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 - https://phabricator.wikimedia.org/T118158#1799184 (10Dzahn) @hashar i did the same for 1.625.2 right now because of further fixes they released [16:14:08] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 / 1.625.2 - https://phabricator.wikimedia.org/T118158#1799197 (10Dzahn) [16:41:36] https://gerrit.wikimedia.org/r/#/c/252443/ [16:41:46] So jenkins was silly and V-1'd it, but then... magic happened? [17:06:57] 10Deployment-Systems, 6Release-Engineering-Team, 6Performance-Team, 6operations, 7HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#1799331 (10bd808) >>! In T103886#1671438, @mmodell wrote: >>>! In T103886#1402775, @faidon wrote: >>... [17:23:21] 10Continuous-Integration-Infrastructure, 6operations: puppet compiler wrongly indicates errors when dealing with subrepositories - https://phabricator.wikimedia.org/T118406#1799356 (10jcrespo) 3NEW [17:53:03] (03PS1) 10JanZerebecki: add all the composer based jobs to ext:ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/252457 (https://phabricator.wikimedia.org/T113049) [17:55:40] (03CR) 10JanZerebecki: [C: 032] add all the composer based jobs to ext:ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/252457 (https://phabricator.wikimedia.org/T113049) (owner: 10JanZerebecki) [17:56:36] greg-g, hi [17:56:51] (03Merged) 10jenkins-bot: add all the composer based jobs to ext:ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/252457 (https://phabricator.wikimedia.org/T113049) (owner: 10JanZerebecki) [17:58:07] !log reloading zuul for e81ee95..acbd851 [17:58:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:04:20] Krenair: hello [18:04:22] greg-g, before making #acl*releng did you know about https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects ? [18:04:31] 6Release-Engineering-Team, 6Phabricator, 7Puppet: phabricator at labs is not up to date - https://phabricator.wikimedia.org/T117441#1799414 (10chasemp) [18:04:41] no, chad made that one [18:04:44] Krenair: "demon created this project." [18:04:45] didn't you make something else? [18:04:46] :P [18:04:54] was it one of those new ruby tags? [18:05:10] the ruby and browser-tests ones had tasks [18:05:11] well, I'll ask him anyway [18:05:17] when he's around [18:05:29] greg-g, yes... did you know about that page anyway? [18:05:33] yes [18:05:37] I know of that page :) [18:05:38] ok [18:05:46] I have read it probably 10 times [18:06:01] well sections of it... [18:06:20] this is the main one I reference: https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_Projects [18:06:26] since it's actual useful information [18:11:57] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1799434 (10JanZerebecki) I'm running into T103346 with this. [18:43:47] 10Continuous-Integration-Infrastructure, 7Jenkins, 7WorkType-Maintenance: Upgrade Jenkins to LTS 1.625.1 - https://phabricator.wikimedia.org/T118157#1799506 (10Dzahn) P.S. Uploaded 1.625.**2** and installed it on gallium today as a follow-up. [18:45:31] 10Continuous-Integration-Infrastructure, 7Jenkins, 7WorkType-Maintenance: Upgrade Jenkins to LTS 1.625.1/1.6.25.2 - https://phabricator.wikimedia.org/T118157#1799508 (10Dzahn) [18:45:51] 10Continuous-Integration-Infrastructure, 7Jenkins, 7WorkType-Maintenance: Upgrade Jenkins to LTS 1.625.1/1.625.2 - https://phabricator.wikimedia.org/T118157#1792874 (10Dzahn) [19:14:44] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins, 7WorkType-Maintenance: Please refresh Jenkins package on apt.wikimedia.org to 1.625.1 / 1.625.2 - https://phabricator.wikimedia.org/T118158#1799543 (10hashar) The pre announce of the security release was the reason to bump the LTS version. Th... [20:04:35] 6Release-Engineering-Team, 6Performance-Team, 3Discovery-Cirrus-Sprint, 7WorkType-Maintenance: Investigate loadEventEnd / domComplete regression as of 2015-10-13 - https://phabricator.wikimedia.org/T116137#1799662 (10EBernhardson) This has been redeployed in wed morning SWAT (~16:45 UTC), i looked at the l... [20:20:07] 6Release-Engineering-Team, 6Performance-Team, 3Discovery-Cirrus-Sprint, 7WorkType-Maintenance: Investigate loadEventEnd / domComplete regression as of 2015-10-13 - https://phabricator.wikimedia.org/T116137#1799715 (10Krinkle) [https://grafana.wikimedia.org/dashboard/db/performance-metrics?panelId=8&fullscr... [21:02:20] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms [21:18:29] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1799860 (10JanZerebecki) Done, when those are merged: https://gerrit.wikimedia.org/r/#/c/252508/ htt... [21:18:54] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1799862 (10JanZerebecki) a:3JanZerebecki [21:25:45] 10Beta-Cluster-Infrastructure: deployment-bastion puppet runs failing due to add_ip6_mapped.pp template parsing error - https://phabricator.wikimedia.org/T118422#1799873 (10thcipriani) 3NEW [21:55:02] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [22:03:52] 10Beta-Cluster-Infrastructure: deployment-bastion puppet runs failing due to add_ip6_mapped.pp template parsing error - https://phabricator.wikimedia.org/T118422#1799951 (10thcipriani) May be failing due to a lack of the facter `ipaddress6_eth0` variable which is part of the template that can't be parsed on line... [23:00:39] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #322: 04FAILURE in 3 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/322/ [23:17:16] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [23:33:57] greg-g: you around? [23:54:53] PROBLEM - Puppet failure on pmcache is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]