[00:02:59] 10Wikimedia-Apache-configuration, 06Operations: Apache slash expansion should not redirect from HTTPS to HTTP - https://phabricator.wikimedia.org/T95164#2801154 (10Dzahn) a:03Dzahn `monospaced text` [00:03:40] 10Wikimedia-Apache-configuration, 10Continuous-Integration-Infrastructure, 06Operations: Apache slash expansion should not redirect from HTTPS to HTTP - https://phabricator.wikimedia.org/T95164#2801161 (10Dzahn) [00:57:53] 10Traffic, 06Operations, 10ops-esams: Cannot connect to cp3039.mgmt.esams.wmnet:22 - https://phabricator.wikimedia.org/T150879#2801275 (10BBlack) The dmesg from the very-delayed bootup is interesting. Normally we get through the bulk of the bootup messages in under 20s, but the process took closer to 3-4 mi... [01:03:16] 10Traffic, 06Operations, 10ops-esams: Cannot connect to cp3039.mgmt.esams.wmnet:22 - https://phabricator.wikimedia.org/T150879#2801276 (10BBlack) 05Open>03Resolved a:03BBlack In any case, this evening I started with connecting to the serial serial console (all normal, responds to enter keypress with a... [01:59:20] https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/tHUcqnWPt3o [08:29:40] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 10media-storage: Mediawiki thumbnail requests for 0px should result in http 400 not 500 - https://phabricator.wikimedia.org/T147784#2801775 (10Gilles) Looking at https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html the definition of 400 seems more... [11:26:27] moritzm: I've rebooted most of the spares (kernel upgrade), was that a bad idea? [11:26:41] it looks like they're installing stuff instead of rebooting properly [11:32:09] upload done BTW, all non-spare cp* hosts have been rebooted [11:35:08] ema: I think we should just shut them down before they're eventually decommed [11:35:18] in the current state they're just wasting energy [11:36:00] usually the PXE config is only set to one-time-boot, but maybe some of those are accidentally set to PXE permanently [11:39:14] heh I think pretty much all of them actually were set to PXE and now they're getting freshly reinstalled [11:46:52] mhh, mark switched cp3022 to the the standard puppet role in August, that was probably intentional [11:47:13] eee0aa3e in ops/puppet [11:48:34] I've rebooted cp3017 cp3018 cp3016 cp3015 cp3019 cp3013 [11:49:00] 3022 is one of the few I left alone because it wasn't role spare :) [11:55:12] that reminds me that I still need to reply on the mail from Jaime on that topic, doing that now [11:57:19] bblack: what do you think, should we shut down the spares as moritz [11:57:24] *suggested? [12:01:12] I followed up on the ops list, better to discuss it there I think [12:02:52] +1 [12:25:13] https://blog.qualys.com/ssllabs/2016/11/16/announcing-ssl-labs-grading-changes-for-2017 [13:46:08] about time :) [13:46:19] we're still A+ or beyond by their 2017 grading changes :) [13:46:36] ema: yeah I could care less if they're shut down, I think it's more a policy and/or ops-esams thing [13:46:54] I don't see us reusing them [14:57:51] bblack, moritzm: time to reboot the lvs servers? I can begin with the secondaries [14:58:25] sounds good to me, they're all upgraded to the latest jessie kernel [15:00:48] yeah [15:52:36] secondaries done, starting with the primaries now [16:22:53] 10netops, 06Operations, 10hardware-requests, 10ops-eqiad, 13Patch-For-Review: Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium - https://phabricator.wikimedia.org/T149829#2802776 (10Cmjohnson) [16:38:20] ema: before you do lvs2002 [16:39:19] ema: papaul asking about doing some bios reflash maintenance on it that needs downtime [16:39:44] bblack: ah, right on time, I just did 2001 :) [16:39:47] ema: https://phabricator.wikimedia.org/T126321 [16:40:07] awesome [17:18:04] all lvs hosts rebooted with the new kernel except for lvs2002 (papaul is working on it) [17:36:21] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2803028 (10GWicke) [18:13:51] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2803153 (10Fjalapeno) @Anomie thanks… The explanation makes sense. @GWicke and I spoke at the Reading Services meeting and we are good to go - the conte... [18:40:00] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2803304 (10GWicke) In the same conversation, @Pchelolo brought up a potential issue around image format conversion / selection. We do have a need to suppo... [19:42:50] 10Traffic, 06DC-Ops, 06Operations, 10ops-codfw: lvs2002 Embedded Flash/SD-CARD iLO errors - https://phabricator.wikimedia.org/T126321#2803055 (10Papaul) I clear the log and will leave this task open for now. Before the system update {F4735526} {F4735528} After the system update {F4735607} {F4735609} [20:10:12] 10Traffic, 06Multimedia, 06Operations, 06Performance-Team, and 4 others: Thumb API: Varnish / CDN questions - https://phabricator.wikimedia.org/T150673#2803660 (10Legoktm)