[00:00:10] 3Beta-Cluster: 404 handler for bits.beta.wmflabs.org serves unparsed php - https://phabricator.wikimedia.org/T76810 (10bd808) 3NEW p:3Normal [00:08:22] * marxarelli wishes limechat had some sort of auto-away/auto-return feature so it didn't seem like he spends all day lunching [00:08:28] * marxarelli *nom* [00:09:21] marxarelli: heh :) ZNC + autoaway :) [00:09:26] changes your nick when you close your laptop [00:09:41] <^d> I've been known to leave my nick to |lunch for days. [00:09:44] <^d> Or |zzz [00:09:56] YuviPanda: oh, nice. does it do that on sleep as well? [00:10:51] marxarelli: yeah. as soon as limechat loses connection to ZNC - no network, laptop closed, whatever, you can nick change [00:12:13] YuviPanda: cool. i'll have to see about getting that installed on my trusty little mini-itx server [00:12:47] :) [00:13:32] * marxarelli remembers when 1TB storage was "cool" [00:17:41] marxarelli: heh :) I just looked up the specs for our new labs boxes some time ago. 384G of RAM on each box.... [00:17:48] I remember when 384G of HDD was cool [00:18:48] * chrismcmahon is just going to cry in a corner [00:20:03] :-) [00:20:27] chrismcmahon: I bet we can find a CPU with cache more than HDD space you remember being cool [00:20:50] 3 years on and I still boggle from time to time at the sheer size of Wikipedia [00:20:54] :D [00:23:59] <^d> YuviPanda: How many of those machines do you get? :) [00:24:09] <^d> 384G x 1 machine isn't very exciting ram! [00:24:13] ^d: 3 new ones, which match the specs of our old 6 ones [00:24:39] <^d> 1.15TB, fun. [00:24:48] <^d> http://ganglia.wikimedia.org/latest/stacked.php?m=mem_total&c=Elasticsearch%20cluster%20eqiad&r=hour&st=1417738977&host_regex= hehe :) [00:24:56] YuviPanda: When I was a senior in college I bought 8M (yup megabytes) of ram to put in my desktop taking it to 16M total. That cost me $800 US and made my desktop the computer on campus with the most ram. [00:25:28] ah 1995 [00:25:35] <^d> bd808: I was 7. [00:26:02] ^d: got you beat, I was 4. I probably wasn't even self aware. Such an accomplishment [00:26:04] * bd808 goes to find his walker [00:26:22] bd808: my first ever machine was a celeron that came with 16M of RAM and then upgraded to 32M at some point [00:26:28] I was 31 :-) [00:26:33] I remember getting 64M at some point and feeling very... very... powerful [00:26:42] just about to quit the music business [00:26:42] * ^d gets off people's lawns [00:27:03] chrismcmahon: heh, I should try to spend time learning a skill that's not programming at some point. [00:27:26] YuviPanda: I spent a decade getting paid to play bass in rock'n'roll bands. I recommend it. [00:27:39] chrismcmahon: heh, with my RSI that's probably out [00:27:39] i'm with ya, bd808. 8M ram made Coaster fly! (https://en.wikipedia.org/wiki/List_of_roller_coaster_related_video_games) [00:28:47] heh. I needed it for the compiler I was writing at the time. Adding 8M cut my build time from 6 hours to 30 minutes. [00:29:17] I would have bought more but I couldn't find a motherboard that would slot more than 16 DIMMS [00:29:21] I needed to sell records on the World Wide Web. :-) [00:29:39] * YuviPanda feels very uncool now [00:30:05] chrismcmahon: Nice. My first commercial website client was selling wildflower seeds [00:30:12] chrismcmahon: bam? http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page?debug=true [00:30:13] YuviPanda: trust us, you are killing it. [00:30:40] chrismcmahon: I was wondering why my new check kept showing it as down... :) [00:31:05] so... beta is down? [00:31:06] bd808: ok, ok. move to Florida already... [00:31:10] yikes [00:31:38] or Reno, which seems to be the new Florida [00:31:40] beta is way way down [00:33:21] chrismcmahon: globaluserpage sounds like a legoktm thing [00:33:32] hi [00:33:36] that's me [00:33:38] yeah, looking... [00:33:55] ahhh [00:33:56] one sec [00:33:59] I know why [00:34:44] sorry! [00:35:04] chrismcmahon: YuviPanda: https://gerrit.wikimedia.org/r/177709 [00:35:42] legoktm: merged [00:35:51] how long does it take to get it on bc? [00:36:18] 15m or so I think [00:36:37] YuviPanda: if nothing's broken and no l10n updates, 3 minutes. [00:36:46] ah, ok [00:36:57] depending on what else is being updated [00:37:16] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad runs every 10 minutes [00:37:37] but can get backed up is there is a long scap running already [00:37:40] my information is out of date it seems :-) at one point that was 3 [00:37:45] heh :) [00:38:06] config changes run as soon as the commit is merged [00:38:17] chrismcmahon: while looking at this I realized that the current check will only catch the worst of errors :) I've fixed them to actually work properly now [00:38:41] sorry about that! [00:39:27] legoktm: As soon as it merges, go to https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ and hit the "build now" link [00:39:39] You'll need to auth with your ldap creds [00:40:11] * legoktm pressed [00:40:26] brrr, now my fixed shinken code won't get to complain here [00:40:28] jeez guys [00:40:36] don't be so quick! [00:40:43] YuviPanda: it's not fixed yet [00:40:48] but in process [00:40:49] YuviPanda: one more, https://gerrit.wikimedia.org/r/177712 (won't break anything without it) :P [00:41:36] OK, I'm done, see you tomorrow [00:41:51] it's loading ok for me now (debug=true still gives the backtrace) [00:42:11] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 400 Bad Request - string '/wiki/Main_Page?debug=true' not found on 'http://en.wikipedia.beta.wmflabs.org:80-u' - 684 bytes in 0.004 second response time [00:43:07] * legoktm hugs shinken-wm [00:43:09] grumble grumble typo [00:47:13] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 53492 bytes in 1.996 second response time [00:47:17] legoktm: ^ \o/ [00:47:27] :D [00:52:22] looks like I'll refactor check_commands before doing bits [00:52:23] mmm [00:52:24] ok [00:53:22] !log mediawiki02 instance is low on space, /tmp has lots of... pngs? [00:53:25] Logged the message, Master [00:53:52] bd808: Reedy ^ can I delete those? [00:54:06] yeah [00:54:10] image scaling presumably [00:54:11] *nod* [00:54:53] !log cleared out pngs from mediawiki02 to kill low space warning [00:54:56] Logged the message, Master [00:55:05] actually [00:55:06] that didn't help [00:55:45] but... I think I should sleep. [01:01:54] !log Deleted a ton of jeprof.*.heap files from deployment-mediawiki02:/ [01:01:56] Logged the message, Master [01:10:10] RECOVERY - Free space - all mounts on deployment-mediawiki02 is OK: OK: All targets OK [01:30:18] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#820318 (10Aklapper) p:5Normal>3Low [01:35:19] Project beta-scap-eqiad build #32482: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32482/ [01:50:18] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #319: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/319/ [01:55:07] Yippee, build fixed! [01:55:08] Project beta-scap-eqiad build #32484: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32484/ [02:16:55] Yippee, build fixed! [02:16:56] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #343: FIXED in 1 hr 25 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/343/ [03:34:53] Yippee, build fixed! [03:34:53] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #116: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/116/ [05:19:01] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #225: FAILURE in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/225/ [05:57:16] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #170: FAILURE in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/170/ [06:12:04] Yippee, build fixed! [06:12:04] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #342: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/342/ [06:15:54] Yippee, build fixed! [06:15:55] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #364: FIXED in 56 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/364/ [06:20:46] Yippee, build fixed! [06:20:46] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #299: FIXED in 4 min 51 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/299/ [06:34:23] Yippee, build fixed! [06:34:24] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #207: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/207/ [06:37:18] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #301: FAILURE in 2 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/301/ [06:39:15] Yippee, build fixed! [06:39:15] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #267: FIXED in 1 min 56 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/267/ [06:47:13] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:50:36] Yippee, build fixed! [06:50:36] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #449: FIXED in 1 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/449/ [07:12:11] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:55:25] Project beta-scap-eqiad build #32524: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32524/ [08:08:08] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#820756 (10Qgil) >>! In T75729#819256, @greg wrote: > Just as some teams use projects like https://phabricator.wikimedia.org/tag/ect-january-2015/, other teams should have the option of creating qua... [08:08:25] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#820758 (10Qgil) [08:15:43] Yippee, build fixed! [08:15:44] Project beta-scap-eqiad build #32526: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32526/ [08:50:02] Yippee, build fixed! [08:50:03] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #329: FIXED in 1 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/329/ [10:08:37] 3Continuous-Integration: Config changes to deployment prep are not deployed - https://phabricator.wikimedia.org/T76714#820949 (10hashar) [10:24:47] (03CR) 10Hashar: [C: 032] Split the first bullet list into three: MW, Front-End and Operations [integration/docroot] - 10https://gerrit.wikimedia.org/r/177571 (owner: 10Jforrester) [10:24:50] (03Merged) 10jenkins-bot: Split the first bullet list into three: MW, Front-End and Operations [integration/docroot] - 10https://gerrit.wikimedia.org/r/177571 (owner: 10Jforrester) [10:34:30] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #171: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/171/ [10:37:57] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #198: FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/198/ [10:43:12] 3Continuous-Integration: Install pnmtojpeg and ddjvu on UbuntuTrusty contintLabsSlave hosts. - https://phabricator.wikimedia.org/T76661#820988 (10hashar) [10:50:35] !log applying mediawiki::multimedia class on contint slaves ( https://phabricator.wikimedia.org/T76661 | https://gerrit.wikimedia.org/r/#/c/177770/ ) [10:50:40] Logged the message, Master [11:05:29] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#821022 (10hashar) [11:06:33] 3Continuous-Integration: Install pnmtojpeg and ddjvu on UbuntuTrusty contintLabsSlave hosts. - https://phabricator.wikimedia.org/T76661#821025 (10hashar) https://gerrit.wikimedia.org/r/177770 has been applied on the contint puppetmaster. So you can get the job to run on Precise labs slaves using: `contintLabsSl... [11:07:05] 3Continuous-Integration: Install pnmtojpeg and ddjvu on UbuntuTrusty contintLabsSlave hosts. - https://phabricator.wikimedia.org/T76661#821026 (10hashar) p:5Triage>3Normal [11:14:53] Project beta-scap-eqiad build #32542: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32542/ [11:20:56] Yippee, build fixed! [11:20:56] Project beta-scap-eqiad build #32543: FIXED in 4 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32543/ [11:24:04] Yippee, build fixed! [11:24:04] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #326: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/326/ [11:40:39] (03PS1) 10Krinkle: doc: Remove stray parenthesis [integration/docroot] - 10https://gerrit.wikimedia.org/r/177777 [11:40:43] (03CR) 10Krinkle: "Fixed in I2cfb6370b3002026d8a229f30b9853df8f4bdd12." (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/177571 (owner: 10Jforrester) [11:40:51] (03CR) 10Krinkle: [C: 032] doc: Remove stray parenthesis [integration/docroot] - 10https://gerrit.wikimedia.org/r/177777 (owner: 10Krinkle) [11:40:55] (03Merged) 10jenkins-bot: doc: Remove stray parenthesis [integration/docroot] - 10https://gerrit.wikimedia.org/r/177777 (owner: 10Krinkle) [11:46:35] Yippee, build fixed! [11:46:36] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #320: FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/320/ [11:54:07] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #344: FAILURE in 1 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/344/ [11:56:25] (03PS2) 10Stan: correct bracket spacing per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176341 [11:56:27] (03PS2) 10Stan: remove redundant selfs (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176340 [11:56:29] (03PS2) 10Stan: use english special globals per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176342 [11:56:31] (03PS2) 10Stan: change delimiter for % per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176338 [11:56:33] (03PS2) 10Stan: remove empty rubocop_todo [selenium] - 10https://gerrit.wikimedia.org/r/176345 [11:56:36] (03PS2) 10Stan: disable global cop around globals that can't easily be removed [selenium] - 10https://gerrit.wikimedia.org/r/176344 [12:19:59] (03PS3) 10Krinkle: jsduck: Remove redundant hack that searched jsduck.log for "Warning" [integration/config] - 10https://gerrit.wikimedia.org/r/174619 (https://bugzilla.wikimedia.org/55668) [12:24:40] (03CR) 10jenkins-bot: [V: 04-1] jsduck: Remove redundant hack that searched jsduck.log for "Warning" [integration/config] - 10https://gerrit.wikimedia.org/r/174619 (https://bugzilla.wikimedia.org/55668) (owner: 10Krinkle) [12:30:28] 3Continuous-Integration: integration-config-tox-py27 fails existing master when repos get deleted from Gerrit - https://phabricator.wikimedia.org/T76853#821116 (10Krinkle) [12:30:54] (03PS1) 10Krinkle: Make integration-config-tox-py27 non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/177783 [12:31:39] (03CR) 10jenkins-bot: [V: 04-1] Make integration-config-tox-py27 non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/177783 (owner: 10Krinkle) [12:31:48] (03CR) 10Krinkle: [C: 032 V: 032] Make integration-config-tox-py27 non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/177783 (owner: 10Krinkle) [12:32:52] (03PS4) 10Krinkle: jsduck: Remove redundant hack that searched jsduck.log for "Warning" [integration/config] - 10https://gerrit.wikimedia.org/r/174619 (https://bugzilla.wikimedia.org/55668) [12:51:47] (03CR) 10Zfilipin: [C: 031] correct bracket spacing per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176341 (owner: 10Stan) [12:55:00] (03CR) 10Zfilipin: [C: 04-1] remove redundant selfs (per rubocop) (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176340 (owner: 10Stan) [12:57:11] (03CR) 10Zfilipin: [C: 031] use english special globals per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176342 (owner: 10Stan) [12:59:00] (03CR) 10Zfilipin: [C: 04-1] change delimiter for % per rubocop (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176338 (owner: 10Stan) [13:00:28] (03CR) 10Zfilipin: "Is this patch set just a rebase? I do not see anything different, but the commit message." [selenium] - 10https://gerrit.wikimedia.org/r/176345 (owner: 10Stan) [13:01:42] (03CR) 10Zfilipin: "Is this just a rebase? Nothing changed in the files, just the commit message." [selenium] - 10https://gerrit.wikimedia.org/r/176344 (owner: 10Stan) [13:11:53] 3Continuous-Integration: integration-config-tox-py27 fails existing master when repos get deleted from Gerrit - https://phabricator.wikimedia.org/T76853#821203 (10hashar) [13:16:49] (03PS1) 10Hashar: Revert "Make integration-config-tox-py27 non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/177787 [13:17:20] (03CR) 10Hashar: [C: 032 V: 032] "If the tests are broken lets fix them but do not ignore them!" [integration/config] - 10https://gerrit.wikimedia.org/r/177787 (owner: 10Hashar) [13:18:21] (03CR) 10Krinkle: "I'm not sure how much the code changed since Dduvall and I tested in labs, but from what I recall bundler was not used from the shared dir" [integration/config] - 10https://gerrit.wikimedia.org/r/176818 (owner: 10Dduvall) [13:18:50] (03Merged) 10jenkins-bot: Revert "Make integration-config-tox-py27 non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/177787 (owner: 10Hashar) [13:21:13] 3Continuous-Integration: integration-config-tox-py27 fails existing master when repos get deleted from Gerrit - https://phabricator.wikimedia.org/T76853#821212 (10hashar) [13:23:30] (03PS1) 10Hashar: Remove FacebookOpenGraph [integration/config] - 10https://gerrit.wikimedia.org/r/177788 [13:24:48] 3Continuous-Integration: integration-config-tox-py27 fails existing master when repos get deleted from Gerrit - https://phabricator.wikimedia.org/T76853#821218 (10hashar) [13:27:33] (03CR) 10Hashar: [C: 032] Remove FacebookOpenGraph [integration/config] - 10https://gerrit.wikimedia.org/r/177788 (owner: 10Hashar) [13:31:20] (03Merged) 10jenkins-bot: Remove FacebookOpenGraph [integration/config] - 10https://gerrit.wikimedia.org/r/177788 (owner: 10Hashar) [14:00:37] (03PS2) 10Krinkle: Add customization of publishers for composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/174412 (owner: 10Hashar) [14:00:43] (03CR) 10jenkins-bot: [V: 04-1] Add customization of publishers for composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/174412 (owner: 10Hashar) [14:10:56] (03CR) 10Stan: "yes, only rebased for other changes currently" [selenium] - 10https://gerrit.wikimedia.org/r/176344 (owner: 10Stan) [14:12:15] (03CR) 10Stan: disable global cop around globals that can't easily be removed (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176344 (owner: 10Stan) [14:15:24] Project beta-scap-eqiad build #32561: FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32561/ [14:20:34] (03Abandoned) 10Stan: remove cop disable that no longer matches anything [selenium] - 10https://gerrit.wikimedia.org/r/176343 (owner: 10Stan) [14:22:45] (03CR) 10Stan: change delimiter for % per rubocop (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176338 (owner: 10Stan) [14:24:10] (03Abandoned) 10Stan: cop no longer firing, removing [selenium] - 10https://gerrit.wikimedia.org/r/176339 (owner: 10Stan) [14:30:25] 3Scrum-of-Scrums, Beta-Cluster: Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project - https://phabricator.wikimedia.org/T76086#821295 (10Dzahn) [14:33:40] (03CR) 10Stan: rubocop line length fixes (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/176255 (owner: 10Stan) [14:35:17] Yippee, build fixed! [14:35:17] Project beta-scap-eqiad build #32563: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32563/ [14:35:41] (03CR) 10Stan: ignore cop because is part of module api (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176257 (owner: 10Stan) [14:38:22] (03PS3) 10Stan: add MediaWiki standard rubocop config [selenium] - 10https://gerrit.wikimedia.org/r/176256 [14:38:24] (03PS3) 10Stan: ignore cop because is part of module api [selenium] - 10https://gerrit.wikimedia.org/r/176257 [14:38:27] (03PS3) 10Stan: rubocop and/or style fixes [selenium] - 10https://gerrit.wikimedia.org/r/176258 [14:38:28] (03PS3) 10Stan: use % instead of %Q per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176259 [14:38:30] (03PS3) 10Stan: change block comments to # comments (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176260 [14:38:33] (03PS4) 10Stan: change compact style module definition to nested (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176261 [14:38:34] (03PS3) 10Stan: correct bracket spacing per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176341 [14:38:36] (03PS3) 10Stan: remove redundant selfs (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176340 [14:38:38] (03PS3) 10Stan: rubocop line length fixes [selenium] - 10https://gerrit.wikimedia.org/r/176255 [14:38:40] (03PS2) 10Stan: add basic documentation comments (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176275 [14:38:42] (03PS3) 10Stan: use english special globals per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176342 [14:38:44] (03PS2) 10Stan: remove extra blank line [selenium] - 10https://gerrit.wikimedia.org/r/176277 [14:38:46] (03PS2) 10Stan: add empty lines (per rubocop style) [selenium] - 10https://gerrit.wikimedia.org/r/176276 [14:38:48] (03PS2) 10Stan: use guard clause per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176279 [14:38:50] (03PS3) 10Stan: change delimiter for % per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176338 [14:38:53] (03PS2) 10Stan: remove extra blank line (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176278 [14:38:54] (03PS2) 10Stan: indentation fix [selenium] - 10https://gerrit.wikimedia.org/r/176281 [14:38:56] (03PS2) 10Stan: update hash syntax per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176280 [14:38:58] (03PS2) 10Stan: remove nil comparison per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176283 [14:39:00] (03PS2) 10Stan: disable cop in this instance only [selenium] - 10https://gerrit.wikimedia.org/r/176282 [14:39:03] (03PS3) 10Stan: disable global cop around globals that can't easily be removed [selenium] - 10https://gerrit.wikimedia.org/r/176344 [14:40:36] 3Continuous-Integration: Jenkins: Generate KSS docs from LESS and publish to doc.wikimedia.org - https://phabricator.wikimedia.org/T60620#821305 (10Prtksxna) [14:44:21] manybubbles: I am in the hangout [14:49:09] (03Abandoned) 10Stan: remove empty rubocop_todo [selenium] - 10https://gerrit.wikimedia.org/r/176345 (owner: 10Stan) [14:57:33] manybubbles: looks like there are connection problems [14:57:35] you froze [14:57:37] yeah [14:57:42] I reconnected [14:57:53] I see you nwo [15:19:13] manybubbles: http://www.mediawiki.org/wiki/Manual:Coding_conventions/Ruby#Base_configuration [15:20:23] 3Continuous-Integration: Jenkins: Generate KSS docs from LESS and publish to doc.wikimedia.org - https://phabricator.wikimedia.org/T60620#821444 (10Chad) [16:51:18] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#821563 (10greg) (everything else is fine) >>! In T75729#820756, @Qgil wrote: > PS: If we create quarterly sprints, I would still consider using the Gregorian / natural numbers for quarters. Most h... [17:00:08] Yippee, build fixed! [17:00:09] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #63: FIXED in 2 hr 38 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/63/ [17:21:41] (03CR) 10Krinkle: [C: 032] "Pushed to Jenkins." [integration/config] - 10https://gerrit.wikimedia.org/r/174619 (https://bugzilla.wikimedia.org/55668) (owner: 10Krinkle) [17:35:30] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#821612 (10Qgil) #Jan-Mar-2015 :) [18:21:41] (03CR) 10EBernhardson: "ping" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [18:23:57] (03CR) 10EBernhardson: "Will self merge in another month with no feedback. Either this is a coding convention we care about and we should assert it with tests, or" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [18:29:49] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#821751 (10dduvall) @zeljkofilipin, (Gah! Sorry, I noticed this task a bit late. I have to figure out a better filter for phab notifications...) It's definitely going to make merging difficult. I hate to say it, bu... [18:30:52] (03PS6) 10EBernhardson: Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 [18:35:49] (03CR) 10Ori.livneh: [C: 032] Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [18:35:54] (03Merged) 10jenkins-bot: Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [18:38:07] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#821765 (10dduvall) Also, I'm wondering if we should perform future RuboCop fixes all in one patch set. The downside to that would be less traceability of each rule's acceptance and corrections; the upside would be... [18:38:44] 3Continuous-Integration: Install pnmtojpeg and ddjvu on UbuntuTrusty contintLabsSlave hosts. - https://phabricator.wikimedia.org/T76661#821766 (10hashar) [18:39:15] 3Continuous-Integration, Wikidata: Delete legacy wikidata-jenkins* slaves - https://phabricator.wikimedia.org/T76333#821769 (10hashar) [18:59:58] Yippee, build fixed! [18:59:58] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #171: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/171/ [19:01:47] (03CR) 10Stan: remove redundant selfs (per rubocop) (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/176340 (owner: 10Stan) [19:14:28] 3Continuous-Integration: Install pnmtojpeg and ddjvu on UbuntuTrusty contintLabsSlave hosts. - https://phabricator.wikimedia.org/T76661#808446 (10cscott) Cool, thanks! I created T76881 to remind me to migrate the last jobs over to node 0.10 now. [19:24:05] 3Continuous-Integration, Code-Review: Connect Differential code review with continuous integration - https://phabricator.wikimedia.org/T31#821908 (10MZMcBride) [19:31:45] PROBLEM - Free space - all mounts on deployment-mediawiki01 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_percentfree.value (<55.56%) [19:31:49] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:33:39] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #337: FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/337/ [19:35:20] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #326: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/326/ [19:37:41] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:38:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:41:14] PROBLEM - Free space - all mounts on deployment-mediawiki02 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.diskspace._var.byte_percentfree.value (<55.56%) [19:56:50] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:25:56] Yippee, build fixed! [20:25:57] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #226: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/226/ [20:35:12] Project beta-scap-eqiad build #32598: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32598/ [20:54:28] 3Beta-Cluster: deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (RSA) to the list of known hosts. - https://phabricator.wikimedia.org/T76907#822562 (10greg) [20:55:13] Yippee, build fixed! [20:55:13] Project beta-scap-eqiad build #32600: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32600/ [21:01:00] 3Beta-Cluster: deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (RSA) to the list of known hosts. - https://phabricator.wikimedia.org/T76907#822581 (10bd808) These dumb failures happen on and off. They are usually transient. I... [21:06:49] Yippee, build fixed! [21:06:49] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #46: FIXED in 2 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/46/ [22:06:43] !log /var full on deployment-mediawiki02 again :((( [22:06:47] Logged the message, Master [22:14:25] 3Beta-Cluster: deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (RSA) to the list of known hosts. - https://phabricator.wikimedia.org/T76907#822694 (10greg) p:5High>3Normal And scap in beta works again, so... [22:16:58] 3Beta-Cluster: deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (RSA) to the list of known hosts. - https://phabricator.wikimedia.org/T76907#822699 (10bd808) [22:21:14] RECOVERY - Free space - all mounts on deployment-mediawiki02 is OK: OK: All targets OK [22:21:25] bd808: did you do that or something else? ^ [22:21:46] !log 1.1G free on deployment-mediawiki02:/var after removing a lot of crap form logs and /var/tmp/cores [22:21:47] Logged the message, Master [22:21:51] greg-g: ^ [22:22:24] I applied `find . -type f -ctime +30 -exec rm {} \+` judiciously [22:23:04] Those 2G log partitions and the prod log rotation stratedgy don't play well together [22:23:44] Also there were 2 hhvm core files that my sweeper script wasn't cleaning up for some reason [22:24:14] but most of that savings ~600M was from rm-ing logs over 30 days old [22:24:35] :/ [22:24:40] atop etc [22:24:42] really I bet we never look at logs more than 5 days old in beta. ever [22:24:56] I'd concur [22:25:07] Oh and I purged the apt cache [22:25:12] that was a bunch [22:25:13] so much else has changed over those five days that it's hard to guess at cause that far back [22:25:49] the only time anyone ever looks at logs older than a week is because there was a breach and they are trying to find out when it started IMO [22:26:10] logs are for now and just a bit ago [22:26:26] logstash! [22:26:29] got it [22:26:35] :) [22:26:45] I would personally logstash all the things [22:26:57] I'd almost recommend /dev/null'ing the non logstash hhvm logs now [22:27:15] hashar has a ticket for that [22:27:24] :) [22:27:24] needs some hiera love [22:27:35] https://phabricator.wikimedia.org/T75262 ? [22:27:49] *nod* [22:28:53] the cores are a bigger pain in beta. the location they are created needs to be configurable and then something needs to shout on this channel when one is dumped [22:29:04] otherwise they are just a giant waste of disk [22:29:39] yep [22:29:51] it's just a tree falling in the woods otherwise [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)