[10:29:06] 10Traffic, 06Operations: Select or Acquire Address Space for Asia Cache DC - https://phabricator.wikimedia.org/T156256#3150178 (10faidon) After a few back and forths and a lot of supporting documentation, we've passed the verification step of the process and we moved on to the next step as of today: > At this... [11:10:12] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3150245 (10MoritzMuehlenhoff) 05Open>03Resolved a:03MoritzMuehlenhoff I think the immediate question as to what happened 2017-03-09 04:00 - 06:00 UTC has been resolved, so... [11:38:39] 10Wikimedia-Apache-configuration, 10ArchCom-RfC, 10Wikidata, 06Services (watching): Canonical data URLs for machine readable page content - https://phabricator.wikimedia.org/T161527#3150312 (10daniel) @Smalyshev the problem is that page titles can contain slashes. If we don't always provide the slot name,... [11:47:35] 10Traffic, 06Operations, 10Pybal: Unhandled pybal ValueError: need more than 1 value to unpack - https://phabricator.wikimedia.org/T143078#3150356 (10ema) 05Open>03Resolved a:03ema Confirmed, upgrading twisted to 16.2.0 fixed this. [12:33:00] 10Traffic, 06Operations, 10Wikimedia-Site-requests, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150458 (10ema) [12:33:16] 10Traffic, 06Operations, 10Wikimedia-Site-requests, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150362 (10Nemo_bis) Same as T162033? [12:33:46] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150463 (10Nemo_bis) [12:33:48] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150362 (10stjn) https://upload.wikimedia.org/wikipedia/commons/thumb/f/f5/Flag_of_Cross_of_Burgund... [12:34:07] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150467 (10Urbanecm) It seems like it. [12:37:22] 10Traffic, 06Operations: Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support) - https://phabricator.wikimedia.org/T147199#3150472 (10MoritzMuehlenhoff) Text looks good to me, but two points: - "(upgrade or use Firefox!)" is somethat confusing since people might think an updated IE would be av... [12:39:21] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150475 (10Urbanecm) [13:06:03] XioNoX: hi :) [13:06:25] hello! [13:08:50] hey XioNoX :) [13:12:39] XioNoX: welcome! [13:12:57] hey! Thanks [13:18:44] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150533 (10Urbanecm) https://upload.wikimedia.org/wikipedia/commons/thumb/1/19/Ambox_currentevent.s... [13:19:50] I'm still in "waking up and acquiring caffiene" mode [13:20:01] lucky you :) [13:20:02] and responding to a few easy things :) [13:20:09] 10Traffic, 06Operations, 10media-storage, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150536 (10ema) Note that the issue is pretty widespread, I'm seeing lots of requests affected by t... [14:01:39] 10netops, 06Operations: asw-a1-codfw spontaneous reboot - https://phabricator.wikimedia.org/T159464#3150744 (10faidon) 05Open>03Resolved a:03faidon Logs didn't show anything and it hasn't happened in a month. Let's resolve for now. [14:26:24] 10Traffic, 06Operations: Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support) - https://phabricator.wikimedia.org/T147199#3150818 (10BBlack) Depending on the context I've been flipping between whether we're talking about just 3DES or both of the non-FS ciphers, sorry. In current weekly stat... [15:30:11] <_joe_> XioNoX: welcome :) [15:31:02] thx! [15:31:22] oh, welcome XioNoX :) [15:38:03] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151086 (10stjn) Another one to the pile: https://upload.wikimedia.org/wikipe... [15:51:35] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150362 (10Joe) @stjn I cannot reproduce your case and we should've fixed the... [15:54:53] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151162 (10Urbanecm) No problem for me. Try to clear cache of your browser. T... [16:03:05] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151195 (10stjn) Yes, the same ERR_CONTENT_DECODING_FAILED even with disabled... [16:19:07] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151248 (10Vachovec1) I can confirm ERR_CONTENT_DECODING_FAILED for https://u... [16:47:44] ema: regarding the commons lang cache stuff, you know if this will be fixed in the next days/weeks? Otherwise i have to adjust the local notice and the hack itself. [16:49:28] <_joe_> ema, bblack: people are still reporting issues ^^ but I can't repro [16:52:29] _joe_: oh so perhaps the VCL hack isn't enough? I imagine that fixing the problem itself in swift would be the way to go then [16:53:52] <_joe_> ema: I think some cases slipped through the ban, maybe [16:56:56] you can tell by the Age I think [16:57:05] (if Age is ~ older than ban-time) [17:58:08] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151626 (10BBlack) [18:01:07] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151609 (10Dzahn) added to ops mailing list [18:02:19] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151609 (10faidon) Added to all 34 network devices (cr, asw/csw, msw, mr, pfw). [18:03:26] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151651 (10BBlack) [18:03:35] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151609 (10BBlack) [18:03:56] haha, so many irc notifications :) [18:05:34] yes, bots spamming IRC for everything that happens is a common pattern here :) [18:07:48] 10Traffic, 06Operations, 10Ops-Access-Requests: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151661 (10BBlack) [18:13:25] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151686 (10Dzahn) added to root@ mail alias (prepare for incoming wave of mails :p) [18:20:33] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151689 (10Dzahn) added Icinga contact in private repo (with just email notification method for now, no phone number / paging just yet) [18:22:58] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151694 (10BBlack) Added to other email aliases in private repo as well: dns-admin, peering, ripe-updates [18:25:17] XioNoX: you'll need mail filters :) [18:25:32] dns-admin is especially fun [18:27:14] :) [18:48:51] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151772 (10BBlack) [18:56:55] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151794 (10ema) Our understanding of the problem so far is that some of our s... [19:22:37] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151981 (10Dzahn) [x] phabricator permissions to see NDA and Ops restricted tickets I did the same steps that were done by @Aklapper in T144496#2601909. - https... [19:23:09] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3151984 (10Dzahn) [19:23:13] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Specific PNG thumbnail delivered as [text/html] instead of [image/png] and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3151985 (10ema) There is probably another type of bug responsible for the ERR... [20:30:51] 10Traffic, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Ops Onboarding for Arzhel Younsi - https://phabricator.wikimedia.org/T162073#3152222 (10ayounsi) @Muehlenhoff, here is my public GPG key for pwstore. ``` -----BEGIN PGP PUBLIC KEY BLOCK----- mQGiBEtGU7gRBADRV1Z96fsxR6riZOD1bL3PVhyKntVakX... [22:00:59] 15:02 < icinga-wm> PROBLEM - Host lvs2002 is DOWN: PING CRITICAL - Packet loss = 100% [22:01:30] bblack: ^ looking at mgmt [22:10:31] i got nothing from console so i powercycled and after a while: [22:10:32] < icinga-wm> RECOVERY - Host lvs2002 is UP: PING OK - Packet loss = 0%, RTA = 36.08 ms [22:10:44] lvs2002 login: [22:11:00] root [22:11:05] :-P [22:18:16] :p i just meant to say it's back up at login as pretending nothing happened :p [22:18:46] it just randomly powered down [22:38:13] password: [22:41:57] 10Traffic, 06Operations, 10ops-codfw: lvs2002 random shut down - https://phabricator.wikimedia.org/T162099#3152568 (10Dzahn)