[00:15:27] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210 (10Ryan Kaldari) 3NEW p:3Unprio s:3normal a:3None http://en.m.wikipedia.beta.wmflabs.org/wiki/Gallery#/editor/0 Editor switcher icon next to Next button doesn't load. W... [00:26:08] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c5 (10Sean Pringle) As part of this outage[1], p50380g50769__wvs2 and p50380g50769__wvs2ds had to be dumped and reloaded into a new db instance. Together they are really bi... [00:29:36] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c1 (10Chris McMahon) adding MMV folks just in case [00:30:23] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c2 (10Greg Grossmeier) Ryan: can you give an example of a working url? I don't know what I'm looking at. :) [00:34:52] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c3 (10Chris McMahon) Created attachment 16572 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16572&action=edit page of images does not render [00:35:53] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c4 (10Ryan Kaldari) Compare with: https://en.m.wikipedia.org/wiki/Cosmic_Tones_for_Mental_Therapy#editor/0 [00:36:37] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c5 (10Chris McMahon) In Chromium I see a Next button but it does nothing. In the latest Chrome the page does not render, see screen shot. [00:38:22] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c6 (10Greg Grossmeier) In Iceweasel on http://en.m.wikipedia.beta.wmflabs.org/wiki/Gallery#/editor/0 I don't see a next button but it renders and there are a lot of images (ie:... [00:44:37] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c7 (10Chris McMahon) Seems to be different behavior for logged-in and anon users? Like Greg, I'm not really sure what should be happening here. [03:33:02] legoktm: ping [03:33:14] yes hello [03:33:52] [05:11:22] (PS1) Jeremyb: add OpenStackManager component to #wikimedia-labs [labs/tools/pywikibugs] - https://gerrit.wikimedia.org/r/162201 [03:33:52] [05:11:40] legoktm: YuviPanda|afk: ^^ [03:34:26] or feel free to just add me to project and I'll do it myself :) [03:35:15] I'll do that after dinner, which I am about to leave for :P [03:35:30] bon appetit [04:08:04] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (20.00%) [04:17:14] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [04:37:14] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (11.11%) WARN: tools.tools-exec-08.diskspace._var.byte_avail.value (100.00%) [04:41:22] (03CR) 10Legoktm: [C: 032] add OpenStackManager component to #wikimedia-labs [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/162201 (owner: 10Jeremyb) [04:42:36] (03CR) 10Legoktm: [V: 032] add OpenStackManager component to #wikimedia-labs [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/162201 (owner: 10Jeremyb) [04:45:04] !log tools.wikibugs deployed https://gerrit.wikimedia.org/r/162201 (add OpenStackManager component to #wikimedia-labs) [04:45:07] Logged the message, Master [04:45:14] jeremyb: ^ [04:47:38] 3MediaWiki extensions / 3OpenStackManager: OpenStackManager: Missing message "<novaresources>" on Special:SpecialPages - 10https://bugzilla.wikimedia.org/67852 (10Kunal Mehta (Legoktm)) [04:47:42] woot. [04:50:10] woot [04:50:33] hah, i wrote that before reading your woot :) [04:51:28] oh, huh, labs-morebots doesn't care if the project exists? [04:51:34] just needs SAL in the right place? [05:33:25] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218 (10wp mirror) 3NEW p:3Unprio s:3major a:3None To whom it may concern, 0) PROBLEM Most of the WMF sites have IPv6 addresses. For example: (shell)$ dig www.mediawiki.org AAAA t... [05:34:10] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218 (10wp mirror) [05:34:10] 3Wikimedia Labs / 3Infrastructure: Enable ipv6 on labs - 10https://bugzilla.wikimedia.org/35947 (10wp mirror) [05:38:07] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218 (10jeremyb) [05:39:52] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218 (10jeremyb) [05:40:07] 3Wikimedia Labs / 3Infrastructure: Enable ipv6 on labs - 10https://bugzilla.wikimedia.org/35947 (10jeremyb) [05:46:22] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218#c1 (10jeremyb) (In reply to wp mirror from comment #0) > (shell)$ dig lists.wikimedia.org [lists.wikimedia.org] AAAA I don't understand this syntax. What is the second "lists.wikimedia.org" on t... [06:30:53] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218#c2 (10wp mirror) (In reply to jeremyb from comment #1) > (In reply to wp mirror from comment #0) > > (shell)$ dig lists.wikimedia.org [lists.wikimedia.org] AAAA > > I don't understand this synta... [06:33:22] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218#c3 (10jeremyb) (In reply to wp mirror from comment #2) ok, great. now we just need bugzilla to allow editing your own comments :-) [06:37:39] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [06:51:19] PROBLEM - ToolLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-wmt.puppetagent.failed_events.value (33.33%) [07:16:46] RECOVERY - ToolLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [10:20:10] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c8 (10Gilles Dubuc) The mobile site doesn't use MMV. It uses a simpler lightbox which has a similar look, but it's totally separate code. [10:49:22] 3Wikimedia Labs / 3Infrastructure: Enable IPv6 on wikitech.wikimedia.org - 10https://bugzilla.wikimedia.org/71218 (10Andre Klapper) p:5Unprio>3Normal [12:27:21] I will let you know when I see andrewbogott around here [12:27:21] @notify andrewbogott [15:07:11] !admin@wikitech https://wikitech.wikimedia.org/wiki/Revenue_Cycle_Management spam spam lovely spam [15:55:51] 3MediaWiki extensions / 3OpenStackManager: Special pages should link to in-wiki documentation - 10https://bugzilla.wikimedia.org/34500#c2 (10Nemo) (In reply to Ryan Lane from comment #0) > Documentation isn't easily findable, or is ignored now. It would be good to > have documentation directly accessible fro... [16:39:46] YuviPanda: ready for some more merges? [17:06:21] andrewbogott: I'm in a bus :( [17:06:44] YuviPanda: ok then -- sorry I missed you earlier [17:08:46] andrewbogott: is OK. Tomorrow? [17:09:17] sure [17:28:23] Stupid computer crashed again [18:05:06] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c6 (10metatron) That's exactly what I feared! Coren wears sackcloth and ashes - that's indicated. A simple announcement *in advance* would have done it – like it happened i... [18:06:08] YuviPanda: i think we may have broken labs phab [18:06:17] YuviPanda: by changing prod phab .. https redirect [18:06:32] but not sure, because ..is it behind proxy or no? [20:29:36] tools.mediawiki-mirror@tools-login:~$ du -shc . [20:29:37] 6.7G . [20:29:41] I hope that's not bad [20:30:02] what is that tool for? [20:30:28] doxygen http://tools.wmflabs.org/mediawiki-mirror/html/dirs.html [20:36:14] neat [20:40:00] https://github.com/hay/wiki-tools/issues/6 [20:42:52] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c11 (10Greg Grossmeier) Is this still an issue? I don't have wikidata going in a vagrant instance, but on beta I can go here without fail: http://wikidata.beta.wmf... [20:52:24] I Guess I can predict the future :P [20:52:36] some people report catscan2 is broken and never answer, perhaps unrelated but this afternoon I needed to move my web service from -03 'cause of oom when starting a python cgi, my lighttpd was far from the limit [20:52:40] log deployment-prep updated OCG to version 48acb8a2031863e35fad9960e48af60a3618def9 [20:52:50] !log deployment-prep updated OCG to version 48acb8a2031863e35fad9960e48af60a3618def9 [20:52:54] Logged the message, Master [20:53:09] https://lists.wikimedia.org/pipermail/labs-l/2014-September/002954.html [21:02:25] 3Wikimedia Labs / 3deployment-prep (beta): [trebuchet] salt commands exceed timeout - 10https://bugzilla.wikimedia.org/70863 (10Greg Grossmeier) p:5Unprio>3Normal [21:33:07] Is https://tools.wmflabs.org/catscan2/catscan2.php down or just half asleep [21:39:09] at least 75 % [22:16:26] Coren, YuviPanda, please read and comment when you have a chance: https://wikitech.wikimedia.org/wiki/Ldap_rename [22:25:03] " (Note: Make sure we know how to do this before any of this begins.)" is full of win. [22:26:09] Yeah, maybe should make that boldface [22:26:34] * Coren politely points out at his last attempt to mess with the opendj java keystore mangled it. [22:26:56] That's… good to know. [22:28:01] I really wish there was a "0. replace opendj with something sane." [22:28:06] Coren, that sounds like you already have half an idea how to do this… want to log into neptunium right now and sort out the other half? [22:28:38] andrewbogott: More tomorrow than today, if you don't mind. I'm about to dinner. [22:28:59] Coren: Sure. [22:30:41] Oh, I know how to do it in principle; the java keystore is a pain but mostly simple to use - it's just that we've also weaved some odd non-standard changes in through puppet (like the cert names aren't the default ones) [22:46:33] 9. run around screaming everything is broken [23:00:24] !log deployment-prep Updated bash with salt [23:00:28] Logged the message, Master [23:01:49] andrewbogott: We should really figure out how to make it so that when you do something via salt across all of labs it also effects the beta hosts. I think Ryan said something once about a way to chain/federate the salt masters. [23:06:58] * Coren wants and eqiad box named unbinilium [23:11:26] this spam is alive for 8 hrs... https://wikitech.wikimedia.org/wiki/Revenue_Cycle_Management [23:15:46] Revi: thanks, deleted [23:31:44] mutante: hey [23:31:50] Still broken? [23:36:16] YuviPanda: hello, i don't know, but we added the http->https redirect in prod [23:36:29] YuviPanda: and labs will use the same apache template, but not be behind misc-web.. right [23:36:37] YuviPanda: unless it is behind instanceproxy and stuff [23:37:05] might not be broken but also not redirect like in prod [23:37:16] because it's based on the x-forwarded header [23:37:24] forwarded-proto that is [23:39:17] We do set up x forwarded proto on the labs proxy [23:39:28] I will take a look tomorrow [23:44:46] YuviPanda: yea, no rush, i just didn't want to advocate labs testing and then we do the same thing and break it with a prod change :p [23:44:59] seems ok for use now though, just wants the redirect