[00:06:12] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:09:50] (03PS3) 10Tim Starling: Revert "Make it compile without C++11" [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/282847 [00:09:56] (03CR) 10Tim Starling: [C: 032 V: 032] Revert "Make it compile without C++11" [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/282847 (owner: 10Tim Starling) [00:10:12] (03PS2) 10Tim Starling: Improve intermediate output [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/282846 [00:10:18] (03CR) 10Tim Starling: [C: 032 V: 032] Improve intermediate output [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/282846 (owner: 10Tim Starling) [00:41:17] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [01:43:44] (03PS2) 10Legoktm: Add Coren (marc@uberbox.org) to CI V+2ers [integration/config] - 10https://gerrit.wikimedia.org/r/282952 (https://phabricator.wikimedia.org/T132345) (owner: 10Jforrester) [01:47:28] (03CR) 10Legoktm: [C: 032] Add Coren (marc@uberbox.org) to CI V+2ers [integration/config] - 10https://gerrit.wikimedia.org/r/282952 (https://phabricator.wikimedia.org/T132345) (owner: 10Jforrester) [01:48:14] (03Merged) 10jenkins-bot: Add Coren (marc@uberbox.org) to CI V+2ers [integration/config] - 10https://gerrit.wikimedia.org/r/282952 (https://phabricator.wikimedia.org/T132345) (owner: 10Jforrester) [01:48:47] !log deploying https://gerrit.wikimedia.org/r/282952 [01:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:59:27] 10Beta-Cluster-Infrastructure: Creating wiki at beta cluster for the Dutch Wikipedia - https://phabricator.wikimedia.org/T118005#2201984 (10Krenair) Who is going to be helping? [03:16:24] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #941: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/941/ [03:19:31] Yippee, build fixed! [03:19:31] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1044: 09FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1044/ [04:22:40] Yippee, build fixed! [04:22:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #780: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/780/ [04:39:15] 10Beta-Cluster-Infrastructure: Change the SSL certificate of beta.wmflabs.org to remove SSL warnings/errors - https://phabricator.wikimedia.org/T129740#2202092 (10Pokefan95) p:05Triage>03Normal [05:26:35] Yippee, build fixed! [05:26:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #756: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/756/ [08:21:07] Yippee, build fixed! [08:21:07] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #920: 09FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/920/ [08:34:00] Yippee, build fixed! [08:34:00] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #939: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/939/ [09:43:31] Yippee, build fixed! [09:43:31] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #778: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/778/ [10:09:11] RECOVERY - Host cache-rsync is UP: PING OK - Packet loss = 0%, RTA = 1.59 ms [10:14:07] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [10:21:08] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202605 (10Paladox) @greg Ive updated the installer to 1.1.2 and now includes VC14 redistributable setup files from Microsoft since those are needed when installing php. I'm not sure... [10:40:09] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202661 (10Aklapper) @Paladox: You need to check the exact license of included files and mention the exact license of included files here. This is not work to be done by others (exce... [10:40:17] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202662 (10Aklapper) p:05Triage>03Lowest [10:41:33] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202663 (10Paladox) @Aklapper according to https://www.microsoft.com/en-gb/download/details.aspx?id=48145 it doesn't show a license. [10:43:44] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202666 (10Aklapper) If there is no information I'm afraid you will have to contact Microsoft to find out yourself, so you yourself will be on the safe side, legally speaking. [10:44:29] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2183662 (10Reedy) >>! In T131939#2202663, @Paladox wrote: > @Aklapper according to https://www.microsoft.com/en-gb/download/details.aspx?id=48145 it doesn't show a license. https://... [10:47:21] 05Gerrit-Migration, 10Differential: Use the arcanist installer for windows i created - https://phabricator.wikimedia.org/T131939#2202679 (10Paladox) Oh thanks for finding that @Reedy looks like I can legally use the redist files, since I'm using the vc_redist files which it says I can. [12:12:00] (03CR) 10Prtksxna: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/282641 (owner: 10Prtksxna) [12:44:13] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [12:54:52] Yippee, build fixed! [12:54:52] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #1018: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/1018/ [13:05:51] 10Beta-Cluster-Infrastructure, 10Browser-Tests-Infrastructure, 15User-zeljkofilipin, 07WorkType-NewFunctionality: Make selenium users use botflags at beta-cluster - https://phabricator.wikimedia.org/T116027#2202997 (10zeljkofilipin) [13:10:42] (03PS1) 10JanZerebecki: wmf: Update Wikidata to wmf/1.27.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/283171 [13:11:43] (03CR) 10JanZerebecki: [C: 032] wmf: Update Wikidata to wmf/1.27.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/283171 (owner: 10JanZerebecki) [13:18:30] (03Merged) 10jenkins-bot: wmf: Update Wikidata to wmf/1.27.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/283171 (owner: 10JanZerebecki) [13:25:42] 10Beta-Cluster-Infrastructure, 06Operations, 07Performance: Need a way to simulate replication lag to test replag issues - https://phabricator.wikimedia.org/T40945#2203091 (10Nikerabbit) [13:46:40] PROBLEM - Puppet staleness on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [14:34:40] Yippee, build fixed! [14:34:41] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #485: 09FIXED in 6 min 39 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/485/ [14:51:28] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 10Traffic, 07HTTPS: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2203375 (10BBlack) As gitblit is on the chopping block for deprecation anyways, my inclination is to go ahead and enable HTTPS for this soon... [14:53:01] 06Release-Engineering-Team, 10Gitblit, 06Operations, 10Traffic, 07HTTPS: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2203384 (10greg) [14:58:15] PROBLEM - Puppet run on deployment-puppetmaster is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:58:30] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:02:55] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:03:09] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:08:29] PROBLEM - Puppet run on integration-slave-trusty-1025 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:14:36] puppet breakage is caused by: WARNING: The following packages cannot be authenticated! [15:14:37] zsh-beta zsh zsh-common [15:14:51] I wonder what happened to the apt keys? [15:25:06] twentyafterfour: I tested some code by allowing us to view all refs like refs/changes and it works. [15:25:22] Would you like me to upload a patch to wikimedia phabricator repo with the change. [15:27:39] paladox: ok [15:31:09] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Make MathML rendering default on BETA wikipedia - https://phabricator.wikimedia.org/T104550#2203613 (10mobrovac) [15:32:47] !log killed https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/9455/ hung at 16% for 30 minutes. [15:34:22] ostriches: random thought of the morning: why don't we rename the "labs" realm in wm-config to "betacluster"? [15:37:36] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:38:06] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:13] blerg. I'm obviously misremembering how to get jenkins to re-try to merge a patch: https://gerrit.wikimedia.org/r/#/c/282994/ How is that done again? [15:38:54] oh. I guess it's working now. I guess just flail at it for a while and it'll work. [15:39:14] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:43:09] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:45:33] twentyafterfour: Ive uploaded the patch to https://gerrit.wikimedia.org/r/283208 [15:45:34] but i think we also need to update DiffusionLowLevelGitRefQuery.php file but i doint know how to hide the commit from diffusion view or how to allow us to do refs/heads without causing master to break. [15:46:59] It detects the commit but says Commit Still Parsing until i update DiffusionLowLevelGitRefQuery.ph [15:59:00] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [16:03:18] RECOVERY - Puppet run on deployment-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:35] RECOVERY - Puppet run on integration-slave-trusty-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:13] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:55] Project beta-update-databases-eqiad build #7828: 04FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7828/ [16:35:27] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:05:29] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [17:07:19] PROBLEM - Puppet run on integration-slave-trusty-1024 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:15:32] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:22] 10Beta-Cluster-Infrastructure, 10Math: Make MathML rendering default on BETA wikipedia - https://phabricator.wikimedia.org/T104550#2204048 (10mobrovac) 05Open>03Resolved Merged and deployed. BetaCluster is using RESTBase / Mathoid with MathML rendering by default now. [17:20:32] Yippee, build fixed! [17:20:32] Project beta-update-databases-eqiad build #7829: 09FIXED in 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7829/ [17:34:07] 06Release-Engineering-Team, 10Gitblit, 06Operations, 10Traffic, 07HTTPS: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204120 (10BBlack) In lieu of anything more-concrete to go on, I've been monitoring the varnishlog live request flow for git.wikimedia.org this mornin... [17:34:42] (testing a shinken change) [17:38:31] twentyafterfour: I think ive found a way to hide refs/changes and refs/meta changes from diffusion gui. [17:52:58] 06Release-Engineering-Team, 10Gitblit, 06Operations, 10Traffic, and 2 others: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204187 (10BBlack) 05Open>03Resolved a:03BBlack Resolving for now, although I suspect this is the most likely of the bunch to trigger some ki... [17:58:40] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #878: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/878/ [18:14:56] 06Release-Engineering-Team, 10Gitblit, 06Operations, 10Traffic, and 2 others: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204324 (10BBlack) For the record: the oddball `Mozilla/8.0 (Windows 2008 SP32 + 3patch)` seems to be making it through the HTTPS redirect just fin... [18:30:47] Yippee, build fixed! [18:30:47] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #656: 09FIXED in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/656/ [18:54:11] Yippee, build fixed! [18:54:12] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #659: 09FIXED in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/659/ [18:58:31] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:11:30] RECOVERY - Puppet run on integration-slave-trusty-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:15] twentyafterfour: Would it be ok if we show the ref changes and meta in gui since we can always later fix that so it is hidden by allowed to view the commit. The way i think im doing it is so that the refs/changes and refs/meta are placed at the bottom so if you have many branches users should not notice it. But it would at least allow us to view open commits and would allow us to not use git.wikimedia.org pretty much any more [19:37:31] PROBLEM - Puppet run on integration-slave-trusty-1025 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:38:27] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:46] 10Beta-Cluster-Infrastructure, 10Monitoring, 13Patch-For-Review, 07Shinken: Monitor keyholder on deployment-bastion - https://phabricator.wikimedia.org/T111064#2204713 (10Krenair) a:03Krenair So when that gets merged, shinken-01 should be running something like this to check: `/usr/lib/nagios/plugins/che... [19:54:58] 10Beta-Cluster-Infrastructure, 10Monitoring, 13Patch-For-Review, 07Shinken: Monitor keyholder on deployment-prep deployment servers - https://phabricator.wikimedia.org/T111064#2204715 (10Krenair) [20:10:14] paladox: I'm not sure if that's the right solution but I will take a look. My main concern is if it slows things down having all those extra refs [20:11:00] twentyafterfour: Ok, i think we can hide them but one thing with that when we want to update the refs for changes again we will need to unhind and re hind afterwards. [20:21:42] twentyafterfour: It shoulden slow anything down, but i think upstream would need to add some kind of support for hidding branches from gui but not totally allowing us to manually go to the link. [20:22:59] 10Beta-Cluster-Infrastructure, 10Monitoring, 13Patch-For-Review, 07Shinken: Monitor keyholder on deployment-prep deployment servers - https://phabricator.wikimedia.org/T111064#1593275 (10hashar) Thank you @Krenair ! [20:23:04] 10Beta-Cluster-Infrastructure, 06Services, 07Puppet, 15User-mobrovac: deployment-(mathoid|sca0[12]|cxserver03) puppet failures "Error: Could not set home on user" due to user being in use by process - https://phabricator.wikimedia.org/T132265#2204755 (10mobrovac) `deployment-(mathoid|sca0[12])` have been f... [20:23:26] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:14] twentyafterfour: See http://www.test-random-wikisaur.tk/diffusion/2/repository/master/ please. [20:24:44] 10Beta-Cluster-Infrastructure, 06Services, 07Puppet, 15User-mobrovac: deployment-(mathoid|sca0[12]|cxserver03) puppet failures "Error: Could not set home on user" due to user being in use by process - https://phabricator.wikimedia.org/T132265#2204772 (10Krenair) Yeah, I can only log in as root there, not m... [20:27:55] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204774 (10hashar) [20:29:12] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204795 (10hashar) I have cleared the obsolete apt configuration files related to web proxy: ``` salt -v 'deployment-cxserver03*' cmd.run 'find /etc -name "*proxy*" -pr... [20:30:44] 10Beta-Cluster-Infrastructure, 06Services, 07Puppet, 15User-mobrovac: deployment-(mathoid|sca0[12]|cxserver03) puppet failures "Error: Could not set home on user" due to user being in use by process - https://phabricator.wikimedia.org/T132265#2192956 (10hashar) For deployment-cxserver03 I have filled {T132... [20:30:47] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204802 (10Krenair) a:03Krenair I'm in the middle of dealing with it @hashar :) [20:32:04] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2204810 (10mobrovac) [20:33:42] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204825 (10Krenair) 05Open>03Resolved Puppet works again, login works. I had to change /etc/puppet/puppet.conf to fix "server = " line to include the `.deployment-p... [20:34:31] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204828 (10hashar) Maybe @Krenair fixed it as part of T132265 . Puppet runs fine now and I have managed to ssh into the instance :-) [20:35:20] 10Beta-Cluster-Infrastructure: deployment-cxserver03 refuses ssh connection (but reachable via salt) - https://phabricator.wikimedia.org/T132626#2204833 (10Krenair) Please do not subscribe AlexMonk-WMF to tasks where that account does not belong [20:35:30] Krenair: thanks for the beta cluster instances fix up :-) [20:35:35] yw [20:35:42] Krenair: sometime the puppet.conf ends up being borked /corrupted for random reason :-\ Never looked at it closley [20:35:54] I am assuming it is just instances that have some outdated conf somehow [20:35:57] yeah, I thought I fixed all the instances with that problem. [20:36:02] But this one appeared [20:36:23] ohh [20:36:42] and Phabricator ends up autocompleting "Krenair" with the AlexMonk-wmf account :-D [20:36:56] ugh [20:37:46] AlexMonk-WMF (Krenair) !=== Krenair (Alex Monk) :D [20:38:14] I guess Phabricator pick the WMF one because it alphabetically comes first [20:38:56] * hashar asks her Majesty the Queen to allow Alex to rename himself "zAlex" [20:39:38] heh [20:41:10] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.29 ms [20:42:16] 10Continuous-Integration-Config, 13Patch-For-Review: Add Coren (marc@uberbox.org) to CI V+2ers - https://phabricator.wikimedia.org/T132345#2204847 (10hashar) 05Open>03Resolved a:03Legoktm Solved by @Legoktm via https://gerrit.wikimedia.org/r/#/c/282952/2/zuul/layout.yaml,cm [20:43:00] Okay, so, *if* we are to trust salt, no more deployment-prep instances have that issue. [20:43:13] (I checked with sudo salt '*' cmd.run 'grep deployment-puppetmaster.eqiad /etc/puppet/puppet.conf') [20:45:21] 10Continuous-Integration-Infrastructure, 10Parsoid: Intermittent jenkins failures with tar and disk related errors. - https://phabricator.wikimedia.org/T128032#2204869 (10hashar) We had a few npm jobs that download Phantom JS which somehow overflowed the little tmpfs disk which is shared by multiple builds. @d... [20:46:37] Krenair: sounds good [20:49:34] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:54] 10Beta-Cluster-Infrastructure, 10Monitoring: Shinken warnings about free space on beta cluster Varnish instances - https://phabricator.wikimedia.org/T76417#2204920 (10Krenair) 05Open>03Resolved Assuming it got resolved at some point [20:53:38] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: hhvm apache fills /var/log/apache2 with access logs - https://phabricator.wikimedia.org/T75262#2204929 (10Krenair) [20:53:41] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2204928 (10Krenair) [20:54:15] 10Beta-Cluster-Infrastructure: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#2204935 (10Krenair) [20:54:17] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#727373 (10Krenair) [20:54:37] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: hhvm apache fills /var/log/apache2 with access logs - https://phabricator.wikimedia.org/T75262#755023 (10ori) You can continue finding and disabling everything that needs disk space, or you can just increase the amount of disk space available to these mach... [20:55:17] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [20:58:41] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2204947 (10Krenair) ```lang=irc Apr 12 20:39:13 Krenair: on deployment-mediawiki01 it looks like the biggest dis... [20:58:57] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Beta-cluster web server fills up /var/log with Apache logs - https://phabricator.wikimedia.org/T75262#2204949 (10Krinkle) [21:08:24] 10Beta-Cluster-Infrastructure, 10VisualEditor: [Regression pre-wmf.22] Math nodes are not getting rendered on Beta cluster - https://phabricator.wikimedia.org/T132620#2204979 (10Ryasmeen) [21:10:22] 10Beta-Cluster-Infrastructure, 10Monitoring, 07Varnish: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#2204983 (10Krenair) Some reporter script running on puppet exec/cron sending data to graphite and shinken monitoring graphite? [21:20:50] Yippee, build fixed! [21:20:50] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #236: 09FIXED in 4 min 48 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/236/ [21:54:06] what's the right phab tag for software (php7 + php-ast extension) to be installed to jenkins job runners? Tag it with 'Jenkins' ? [21:56:42] ebernhardson: ci-infra [21:57:46] 10Continuous-Integration-Infrastructure: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2205159 (10EBernhardson) [21:58:32] 10Continuous-Integration-Infrastructure: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2205146 (10EBernhardson) [21:58:33] legoktm: thanks [22:07:30] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Beta-cluster web server fills up /var/log with Apache logs - https://phabricator.wikimedia.org/T75262#2205257 (10Dzahn) Though, disabling the logs means you dont have to deal with data-retention issues. [22:10:28] RECOVERY - Puppet run on integration-slave-trusty-1024 is OK: OK: Less than 1.00% above the threshold [0.0] [22:11:13] np [23:11:31] Krinkle it works, just that it shows in the branch list so it may make things horrible. But on the plus side it lets you view refs/changes and refs/meta/ changes [23:11:55] I will not review it. Don't tell me :) [23:12:27] Ok [23:15:48] 10Continuous-Integration-Infrastructure: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2205146 (10Legoktm) Are there debs available for php7 and php7-ast that we can use on trusty or jessie? [23:46:06] Krinkle, You said that you did not want to view the link i sent, but i fixed it and can now be viewed publically http://www.test-random-wikisaur.tk/diffusion/2/