[00:33:27] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [00:50:52] wat [00:52:38] bd808: greg-g should I worry ^^? I don't think I've ever seen bastion max out. [00:53:05] .oO(why am I even still on IRC at this hour?) [00:53:24] chrismcmahon: heh. good questions both [00:53:59] var filling up is likely something logging like crazy [00:54:16] * bd808 takes a peek [00:54:28] it was quiet. a little TOO quiet... [00:57:02] meh. nothing bad. just the damn small disk and large logging output from process accounting and atop [00:57:22] shinken should maybe be a little less paranoid [00:57:46] we should have bigger /var paritions [00:58:03] 1.9G is tiny for the crap we log [00:58:20] true dat [00:58:41] I just was just surprised to see it max out bastion instead of some other host [00:59:30] !log deleted 8 oldest atop.log files on deployment-bastion to free some disk on /var [00:59:44] where's my bot! [01:02:58] see ya [01:08:26] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [03:11:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:21:20] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:22:59] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:38:01] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:55:11] Project beta-scap-eqiad build #35089: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35089/ [04:05:13] Project beta-scap-eqiad build #35090: STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35090/ [04:15:27] Yippee, build fixed! [04:15:27] Project beta-scap-eqiad build #35091: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35091/ [04:27:10] Yippee, build fixed! [04:27:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #432: FIXED in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/432/ [04:32:35] Yippee, build fixed! [04:32:35] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #298: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/298/ [04:34:43] Yippee, build fixed! [04:34:44] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #80: FIXED in 2 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/80/ [04:46:08] Yippee, build fixed! [04:46:09] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #377: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/377/ [04:49:30] Yippee, build fixed! [04:49:30] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #191: FIXED in 58 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/191/ [04:54:14] Yippee, build fixed! [04:54:15] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #334: FIXED in 4 min 44 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/334/ [05:07:11] Yippee, build fixed! [05:07:11] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #242: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/242/ [05:17:20] Yippee, build fixed! [05:17:20] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #399: FIXED in 55 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/399/ [05:43:49] Yippee, build fixed! [05:43:49] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #485: FIXED in 1 hr 16 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/485/ [05:56:31] Yippee, build fixed! [05:56:31] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #406: FIXED in 1 hr 10 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/406/ [06:24:58] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #442: FAILURE in 1 hr 20 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/442/ [06:42:00] Yippee, build fixed! [06:42:01] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #240: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/240/ [06:50:27] Yippee, build fixed! [06:50:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #360: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/360/ [06:55:19] 3Release-Engineering, Wikimedia-Labs-wikitech-interface: add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]] - https://phabricator.wikimedia.org/T73165#941281 (10jeremyb) [07:12:04] Yippee, build fixed! [07:12:05] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #362: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/362/ [07:14:09] Yippee, build fixed! [07:14:10] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #79: FIXED in 2 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/79/ [07:16:52] 3Release-Engineering: Performance Testing Cluster - https://phabricator.wikimedia.org/T282#941304 (10jeremyb) [07:27:19] Yippee, build fixed! [07:27:19] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #360: FIXED in 5 min 53 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/360/ [08:22:34] 3Quality-Assurance, MediaWiki-extensions-Flow: Flow reply_moderation browser test has erroneous selector for "3rd comment on the topic" - https://phabricator.wikimedia.org/T85201#941336 (10Spage) p:5Triage>3Low [08:40:31] 3operations, Beta-Cluster: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#941353 (10yuvipanda) The particular outage that triggered this would've either returned 500 or served blank pages (yayPHP?) [09:55:23] 3operations, Beta-Cluster: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#941381 (10Joe) a:3Joe [10:15:02] Project beta-scap-eqiad build #35127: FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35127/ [10:17:12] 3operations, Beta-Cluster: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#941397 (10Joe) I have a patch up for review, https://gerrit.wikimedia.org/r/#/c/181550/ I'll ask Yuvi/Bryan to test this on beta and then merge while I'm on vacation,or I'l... [10:25:12] Project beta-scap-eqiad build #35128: STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35128/ [10:35:29] Yippee, build fixed! [10:35:29] Project beta-scap-eqiad build #35129: FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35129/ [11:52:30] 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941500 (10fgiunchedi) p:5Triage>3High looks like we're close to resolution for this? [11:54:56] 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941506 (10yuvipanda) More like we have a bandaid that works :) 'Solution' I would say is have some way to make the kernel *not* do core dumps at all - bsd does this, but linux I can't find a... [12:09:56] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:39:54] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:28:13] 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941548 (10fgiunchedi) under linux it should be enough to set the process' rlimit_core to 0 to disable core dumps, unless I'm missing something? [13:59:12] 3operations, Beta-Cluster: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#941587 (10fgiunchedi) p:5Triage>3High [14:53:31] (03CR) 10Jarry1250: "hashar, does the format you've used here imply that dependencies of dependencies are not auto-discovered? BEcause TranslateSvg depends on " [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar) [14:53:34] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:53:50] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:59:26] (03PS1) 10Jarry1250: Translate depends on ULS which depends on EventLogging... [integration/config] - 10https://gerrit.wikimedia.org/r/181574 [15:00:58] (03CR) 10Jarry1250: "Have submitted https://gerrit.wikimedia.org/r/#/c/181574/ on that basis" [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar) [15:02:09] (03PS2) 10Jarry1250: Translate depends on ULS which depends on EventLogging... [integration/config] - 10https://gerrit.wikimedia.org/r/181574 [15:23:46] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:00] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:58:05] 3Quality-Assurance, MediaWiki-extensions-Flow: Flow reply_moderation browser test has erroneous selector for "3rd comment on the topic" - https://phabricator.wikimedia.org/T85201#941872 (10Cmcmahon) Thanks for filing this. I noticed this in a test run recently. I meant to fix it right away and forgot. I'll take... [16:01:49] !log test [16:04:01] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:11] for the first time since shinken started I had <50 emails overnight [16:35:06] Project beta-scap-eqiad build #35165: FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35165/ [16:45:13] Project beta-scap-eqiad build #35166: STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35166/ [16:55:13] Yippee, build fixed! [16:55:13] Project beta-scap-eqiad build #35167: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35167/ [17:10:25] (03CR) 10Nikerabbit: [C: 031] "+1 but I'd appreciate if commit message was amended to not spread information which is easy to understand incorrectly." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/181574 (owner: 10Jarry1250) [17:11:56] (03PS3) 10Jarry1250: Translate depends on ULS [integration/config] - 10https://gerrit.wikimedia.org/r/181574 [18:35:34] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #154: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/154/ [18:45:20] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #207: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/207/ [19:05:56] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #372: FAILURE in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/372/ [19:07:58] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #433: FAILURE in 50 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/433/ [19:30:00] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:52:46] hi ryasmeen I have a quick question about tables in VE I'd like to ask you, since we're not pair-programming today. Do you have a couple of minutes on IRC? [20:05:02] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:16] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #407: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/407/ [20:29:33] 3Quality-Assurance, MediaWiki-extensions-Flow: Flow reply_moderation browser test has erroneous selector for "3rd comment on the topic" - https://phabricator.wikimedia.org/T85201#942298 (10Cmcmahon) The more I look at this, the more I think I want to rearrange how the whole FlowPage is structured. Having rece... [21:00:07] Hi chrismcmahon [21:03:12] hi ryasmeen question is about tables locators, one moment... [21:04:13] sure [21:06:14] ryasmeen: in VE when you do Insert/Table, and then inspect-element on any table cell, it turns out that every cell in the table has "rowspan=1/colspan=1", and that seems clearly wrong. http://imgur.com/EoAnj3u [21:07:08] ryasmeen: do you know if the VE people intend to put the correct rows and columns on the table cells? I hesitate to try automating anything about tables without that. [21:07:33] ryasmeen: (and if not, maybe you could ask them to?) [21:07:42] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #351: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/351/ [21:08:51] oh? but why is it wrong? [21:09:44] rowspan and colspan are not ids of any kind for each cell [21:10:10] ryasmeen: because row 1 column 1 has "rowspan=1/colspan=1", row 3 column 3 has "rowspan=1/colspan=1". shoudnt that be "rowspan=3/colspan=3"? [21:10:17] they just mean how many cells each cell are spanning over [21:10:28] horizontally and vertically [21:11:13] if you merge two horizontally adjacent cells and inspect that one, you will get a value 2 for colspan [21:12:32] hmm. ok. it would certainly be useful to be able to identify a particular cell in a table. [21:13:16] * chrismcmahon should go see how the socialtext wiki does tables, they've had tables for 7 or 8 years now [21:38:40] (03CR) 10Hashar: "@Jarry1250 you indeed have to define the whole chain of dependencies which is very mad. The dependency thing was made a year or so again " [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar) [22:02:48] Yippee, build fixed! [22:02:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #207: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/207/ [22:19:03] (03CR) 10Jarry1250: "Yes -- but Translate only "depends" on ULS in JavaScript/RL ways, hence the recent change made the difference. Very grateful if you could " [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar) [22:39:43] Yippee, build fixed! [22:39:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #208: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/208/ [22:54:32] (03PS1) 10Jdlrobson: Publish documentation for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/181693 [22:54:55] Project beta-scap-eqiad build #35202: FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35202/ [23:05:00] Project beta-scap-eqiad build #35203: STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35203/ [23:15:36] Yippee, build fixed! [23:15:36] Project beta-scap-eqiad build #35204: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35204/ [23:22:48] 3Beta-Cluster, MediaWiki-User-login-and-signup: Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - https://phabricator.wikimedia.org/T72145#942595 (10kaldari) @greg: I can confirm this bug is still occurring. [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)