[00:19:33] 10Traffic, 06Operations, 06Zero, 13Patch-For-Review: Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two - https://phabricator.wikimedia.org/T124482#2205612 (10BBlack) https://gerrit.wikimedia.org/r/283364 above does the functional user-facing change. If it's successful without issue... [00:48:34] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2205680 (10MZMcBride) >>! In T132407#2198891, @Milimetric wrote: > There's some debate about this. We haven't used data.wikimedia.org in the past because of the possible confusion with w... [04:26:23] 07Varnish, 10MobileFrontend, 13Patch-For-Review, 03Reading-Web-Sprint-70-Lady-and-the-Trumps, 05WMF-deploy-2016-04-26_(1.27.0-wmf.22): Stop default redirecting Samsung Smart TVs to mobile web - https://phabricator.wikimedia.org/T127021#2205939 (10dr0ptp4kt) [04:27:22] 07Varnish, 10MobileFrontend, 13Patch-For-Review, 03Reading-Web-Sprint-70-Lady-and-the-Trumps, 05WMF-deploy-2016-04-26_(1.27.0-wmf.22): Stop default redirecting Samsung Smart TVs to mobile web - https://phabricator.wikimedia.org/T127021#2029958 (10dr0ptp4kt) 05Open>03Resolved a:03dr0ptp4kt @jdlrobso... [07:24:25] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: varnishkafka logrotate cronspam - https://phabricator.wikimedia.org/T129344#2206044 (10elukey) 05Open>03Resolved [07:48:40] 07Varnish, 10MediaWiki-Vagrant: MediaWiki-Vagrant varnish role fails to provision - https://phabricator.wikimedia.org/T132337#2206068 (10Gilles) 05Open>03Resolved a:03Gilles Indeed, my commit fixed that problem by pointing to the new repo. [08:32:18] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 3 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2206219 (10Jdlrobson) So here are some facts * On pages impacted by the bug, when queried ParserOutput->getTOCEnabled returned f... [08:34:04] 07Varnish, 10MediaWiki-Vagrant: MediaWiki-Vagrant varnish role fails to provision - https://phabricator.wikimedia.org/T132337#2206221 (10phuedx) Ta. [10:53:48] morning [10:59:50] hey bblack! [11:02:25] hey [11:02:45] I guess we're basically ready to convert actual misc cluster systems at this point on the v4 front, just a question of good timing [11:03:03] probably the big codfw test next week doesn't make for good timing heh [11:03:32] and we probably wouldn't finish switching them all today before the weekend either, so IMHO we stall on that till after the codfw-switchover week [11:04:31] bblack: agreed! We also still have to reboot cp* hosts for the 4.4/point release upgrade [11:04:37] in the meantime, something that's become more time-crunchy by the day: we need to get good stats on http2/spdy [11:04:58] in my mind I was kinda hoping to do 4.4 first to make the systemtap part simpler with all on one version [11:05:16] but IMHO we just have to wing it on mixed kernels with two different compiles for now I guess [11:05:46] yup. For the boot-time race condition, this morning I found a (ugly) workaround: sleep .5 in /etc/initramfs-tools/scripts/local-top/mdadm-enhance-your-calm :) [11:06:28] basically get the systemtap thing usable everywhere, and first take a 1h sample across all the caches and aggregate up the results, make sure our test methodology is sane, doesn't cause horrible perf fallout, etc [11:06:29] tried to reproduce the race condition on a qemu as well, but the disks come up too fast [11:07:14] and then do a 24h sample as a stronger baseline data set. maybe automate that process just enough that it's easy to do it two weeks later again on the same day-of-week, [11:07:30] then probably if results are as expected, make the decision to switch [11:09:16] ema: if puppetizing the initramfs calmness (+ triggering initramfs update) is simpler than puppetizing something like rootdelay=, I'd say go conservative with like a full 2-second sleep, and plan to have that fix progressively invade things until we do it for all jessies? [11:09:31] starting with our caches of course [11:10:28] bblack: I'm not sure if rootdelay would work to begin with, the root device *does* show up, it's just degraded [11:10:45] but that's just me guessing [11:10:53] 07HTTPS, 10Traffic, 10Datasets-General-or-Unknown, 06Operations, 13Patch-For-Review: HTTPS redirects for datasets.wikimedia.org - https://phabricator.wikimedia.org/T132463#2206392 (10mforns) @Ottomata We might want to change this to https then: https://github.com/wikimedia/analytics-dashiki/blob/master/s... [11:11:21] ema: yeah I think the question is whether rootdelay would delay even trying to mount the root device, and thus triggering it getting created in a degraded state before the second disk shows up or whatever [11:26:31] 10Traffic, 10Analytics, 06Operations: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2206419 (10elukey) a:05elukey>03None [11:33:22] 10Traffic, 06DC-Ops, 06Operations, 10ops-esams: cp30[34]x hw/firmware/BMC issues - https://phabricator.wikimedia.org/T126062#2206420 (10BBlack) FTR: I think I've done the blacklist hack on a couple more since, but not recorded them here. There was some suggestion elsewhere that we may need an iDRAC firmwa... [11:49:46] 10Traffic, 10Analytics, 06Operations: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2206447 (10BBlack) Should be fixed now. I did: ``` root@neodymium:~# salt -v -t 10 -L `cat exmobile` cmd.run 'rm -f /etc/logrotate.d/varnishkafka-*' ``... [11:50:47] thanks bblack! --^ [12:45:41] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2204871 (10Anomie) PATH_INFO is fine and expected. The supposedly-bad PHP_SELF is probably also ok (at lea... [13:00:45] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=718533 seems similar to T131961 [13:00:45] T131961: Boot time race condition when assembling root raid device on cp1052 - https://phabricator.wikimedia.org/T131961 [13:01:10] nice one stashbot [13:02:01] isn't it supposed to update the ticket though? [13:02:05] oh well [13:03:07] initramfs-tools (0.119) unstable; urgency=medium [13:03:07] . [13:03:08] The "Wait smarter not harder" release [13:03:10] heh [13:03:12] heh [13:03:51] [8402332] local: Use max(30, rootdelay) as timeout for block device to appear [13:03:58] [2e2f753] local: Call local-block boot scripts to prepare additional block devices [13:04:06] right [13:05:18] so perhaps something is not working properly in scripts/local [13:05:59] yeah we're on a later rev than that I think even in stable jessie now [13:06:16] 0.120 [13:06:47] maybe some other related hack we've done at install-time is mucking with the new fixes? [13:07:35] 10Traffic, 10Analytics, 06Operations: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2206663 (10elukey) 05Open>03Resolved [13:17:21] mmmh perhaps the issue is in mdadm instead [13:17:21] /usr/share/initramfs-tools/scripts/local-top/mdadm calls wait_for_udev *after* calling mdadm --assemble [13:19:47] #714155 [n| | ] [mdadm] initramfs-tools: mdadm starts up before all needed devices are available [13:19:53] #770002 [n| | ] [mdadm] initramfs-tools: the mdadm runs arrays when all disks are not yet ready. [13:20:03] #632401 [i|MR|=] [mdadm] [initramfs-tools] software raid1 does not boot [13:20:15] sounds like we're not alone! :) [13:23:52] yup [13:25:07] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2206706 (10BBlack) [13:25:10] 07HTTPS, 10Traffic, 06Operations: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2206705 (10BBlack) [13:25:33] oh [13:25:37] /usr/share/doc/mdadm/FAQ.gz [13:25:50] 26. Why doesn't mdadm find arrays specified in the config file and causes the boot to fail? [13:26:02] Workaround: there is nothing mdadm can or will do against this. Fortunately [13:26:05] though, initramfs provides a method, documented at [13:26:07] http://wiki.debian.org/InitramfsDebug. Please append rootdelay=10 to the [13:26:10] kernel command line and try if the boot now works. [13:27:14] that's from 2009 though... [13:27:25] 07HTTPS, 10Traffic, 06Operations: enable https for mirrors.wikimedia.org - https://phabricator.wikimedia.org/T132450#2206709 (10BBlack) [13:27:27] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2206708 (10BBlack) [13:28:19] 07HTTPS, 10Traffic, 06Operations: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2198925 (10BBlack) [13:30:22] 07HTTPS, 10Traffic, 06Operations: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2206747 (10BBlack) [13:30:39] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2206750 (10BBlack) [13:30:41] 07HTTPS, 10Traffic, 06Operations: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#366296 (10BBlack) [13:31:50] bblack: would it be OK to try reproducing T131961 on unused ex-mobile machines? (eg: cp1046) [13:31:50] T131961: Boot time race condition when assembling root raid device on cp1052 - https://phabricator.wikimedia.org/T131961 [13:31:58] thanks stashbot we got it [13:31:58] paravoid: any opinion on carbon's (ubuntu|apt|mirrors).wikimedia.org lack of HTTPS and such? I'm assuming all 3 separate hostnames are independently-useful, and should work in browsers, and need to be public, etc... [13:32:05] ema: yeah [13:32:40] that and status.wm.o (watchmouse) are basically the last pragmatic blockers for wikimedia.org HSTS-preload [13:32:46] ubuntu.wm.orgu is deprecated in favor of mirrors.wm.org/ubuntu [13:33:00] I set that up when I added debian (instead of adding debian.wikimedia.org etc.) [13:33:24] but ubuntu is probably hardcoded in a bunch of places, like Ubuntu's mirror list [13:33:47] you'd think mirrors and such would all use https these days anyways heh [13:34:01] they are not [13:34:05] yeah :/ [13:34:15] obviously, there's no listener on 443 [13:34:20] and in fact apt needs an extra package to talk https [13:34:24] apt-transport-https [13:34:40] so we cannot do HTTP->HTTPS redirects on these hostnames [13:34:48] for the immediate stuff (unblocking STS-preload) we don't necessarily need redirects/HSTS-header, just working HTTPS at all [13:34:52] but, we could optionally support HTTPS [13:34:54] yeah, that [13:35:05] it's safe to assume that whoever understands HSTS can talk HTTPS :) [13:35:16] technically, we *should* have redirects and HSTS on all of wikimedia.org to quality, but all they really check is the root + www. [13:35:47] s/quality/qualify/ [13:36:06] for the rest, sure, preload browsers will use preload-STS and everyone else will ignore it, which is ok [13:36:29] the important thing is if preload-STS turns on but there are domains like these with no HTTPS listeners, the preloading browsers can't reach the site at all [13:37:36] probably what this means is we'll have to buy 3x more certs and deploy them there (or 1x cert with 3x SAN, either way) [13:37:51] and then there's watchmouse, not even sure what our options are there [13:40:59] none of this mattered months ago because we were blocked on FR's email.donate subdomain stuff with no end in sight. but the end of that finally came [13:41:04] now we're only blocked on our own stuff heh [13:44:02] anything ongoing with cp1046? I can't ssh to it and console com2 doesn't work [13:44:14] 07HTTPS, 10Traffic, 06Operations: Preload STS for wikimedia.org - https://phabricator.wikimedia.org/T132685#2206791 (10BBlack) [13:44:32] 07HTTPS, 10Traffic, 06Operations: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2206806 (10BBlack) [13:44:35] elukey: yep it didn't reboot properly. power cycling now [13:44:35] 07HTTPS, 10Traffic, 06Operations: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2206807 (10BBlack) [13:44:37] 07HTTPS, 10Traffic, 06Operations: Preload STS for wikimedia.org - https://phabricator.wikimedia.org/T132685#2206791 (10BBlack) [13:44:52] ack ema [13:48:41] 10Traffic, 06Operations, 06Zero, 13Patch-For-Review: Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two - https://phabricator.wikimedia.org/T124482#2206829 (10BBlack) Holding on merging the above until after the codfw-switchover week, so as not to create too many overlapping effects... [13:51:54] 10Traffic, 06Operations, 07Performance: missing SPDY coalesce for upload.wm.o for images ref'd in projects' page outputs - https://phabricator.wikimedia.org/T116132#2206839 (10BBlack) On the Zero issues: the latest update from the Zero team is they still have exactly one carrier that cares about the multimed... [13:54:23] alright yes rootdelay seems to work fine. For the record: cp1046 tried assembling the raid device before any sd device was ready after my first reboot attempt [13:56:48] 10Traffic, 06Operations, 07Performance: missing SPDY coalesce for upload.wm.o for images ref'd in projects' page outputs - https://phabricator.wikimedia.org/T116132#2206854 (10BBlack) Also should note (I thought it was mentioned earlier, but apparently not): we're not even sure how we'd structure this at the... [13:57:40] ema: but rootdelay consistently fixes it there? [13:58:13] I've tried only one reboot with rootdelay=5 and it worked [13:58:15] also, have you looked at how hard puppetizing rootdelay properly is? I kind of assumed it would be a mess to inject it into the things update-grub reads, etc [13:58:45] so I see there is a modules/base/manifests/grub.pp doing stuff with /etc/default/grub and update-grub [13:58:54] we probably want it at install-time too, so it may be better put in the installer and then salt-hack the same effects onto current hosts [13:59:00] I donno [13:59:32] (or just do it separately in both in a way that doesn't conflict) [14:03:33] on the TLS AEAD progress bars: the slow death slope of ecdhe-ecdsa-aes128-sha (much of which is from IE before 11/Edge) in recent times has reversed course a little bit [14:03:55] I wonder if this is backlash against recent Windows auto-upgrades, with people re-downgrading [14:05:15] (that cipher bucket has other causes too, though, like some older android, safari, and java versions) [14:05:17] yeah, grub.pp is a mess though because augeas didn't work pre-jessie [14:05:38] I'd go with the initramfs option probably [14:07:24] paravoid: because it's easier to puppetize or for other reasons? [14:07:37] and yes I agree it's easier to puppetize :) [14:08:03] 10Traffic, 06Operations, 07Performance: missing SPDY coalesce for upload.wm.o for images ref'd in projects' page outputs - https://phabricator.wikimedia.org/T116132#2206896 (10BBlack) 05Open>03stalled [14:08:30] yeah, easier to puppetize [14:11:02] right, I'll first check if that also works consistently on cp1046 though [14:17:33] ema: you might try cp3022 too (with possible package/kernel updates first?) it's decommed but still alive (kinda like ex-mobile, but actually scheduled to really decom instead of reuse) [14:17:44] and it has spinny disks instead of SSDs, which might show the problem easier [14:18:01] alright! [14:18:15] encouraging results on cp1046 so far for the initramfs method too [14:18:38] cool [14:22:25] 07HTTPS, 10Traffic, 10Datasets-General-or-Unknown, 06Operations, 13Patch-For-Review: HTTPS redirects for datasets.wikimedia.org - https://phabricator.wikimedia.org/T132463#2206943 (10Milimetric) done https://gerrit.wikimedia.org/r/#/c/283446/ [14:23:19] bblack: confd is activating instead of active on cp1008 according to icinga [14:24:35] yeah I donno what's up with that, I assumed earlier something _joe_ was working on [14:25:48] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2206951 (10Milimetric) the pageview API is running on wikimedia.org, that's the prod cluster. This task right now is about having a production domain for reports like this: https://brows... [14:28:40] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2206958 (10Milimetric) > Potentially tangentially: I'm unclear what the distinction between Labs and production is when the Wikimedia Foundation is running/operating both. Are there other... [14:33:33] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2206983 (10BBlack) >>! In T132407#2205680, @MZMcBride wrote: > Potentially tangentially: I'm unclear what the distinction between Labs and production is when the Wikimedia Foundation is r... [14:39:52] ema: oh another thing we know from working on ulsfo hardware maint yesterday: if you want to reproduce harder: the problem is much more likely on cold boot than warm boot. [14:40:16] but you'd have to shutdown -h and then go into the mgmt console and racadm serveraction powerup to bring it back online [14:43:04] bblack: will try! cp3022 is doing the right thing [14:54:02] 10netops, 06Operations: turn-up/implement zayo wave (579171) for ulsfo-codfw - https://phabricator.wikimedia.org/T122885#2207068 (10mark) [15:02:30] I've scheduled a few hours of downtime for cp3022, shutting down now [15:06:24] cold booting also works fine [15:12:10] awesome [15:12:41] puppetizing now [16:03:26] bblack: I'm not 100% sure whether the problem is jessie-specific or not [16:07:28] I'd personally would have created an initramfs module that has a define for setting up hooks, but that looks okay to me to :) [16:07:31] *too [16:14:44] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2207372 (10Nuria) [16:15:13] 10netops, 10Analytics-Cluster, 06Operations, 10hardware-requests: setup/deploy server analytics1003/WMF4541 - https://phabricator.wikimedia.org/T130840#2207374 (10faidon) OK, I debugged this some more, and this was caused by a Juniper bug and one that I vaguelly recall experiencing before. The issue was th... [16:16:59] paravoid: oh yeah that's a nice idea [16:21:39] paravoid: so perhaps a define taking $type and $name as parameters, allowing to set type to either hooks or scripts? [16:21:56] I guess :) [16:22:07] yeah, or something like that [16:22:49] but how to specify the script/hook contents themselves? As a simple string? [16:28:58] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2207393 (10NahidSultan) Another one: https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Bizness.w... [16:32:04] 10netops, 10Analytics-Cluster, 06Operations, 10hardware-requests: setup/deploy server analytics1003/WMF4541 - https://phabricator.wikimedia.org/T130840#2207399 (10faidon) The network config was reverted and it still works, so it should be final now. However, the installer failed at the last step with: ```... [16:40:27] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2207408 (10Milimetric) @BBlack: just in case there's some concern about what the purpose of analytics.wikimedia.org is. We will never use it to proxy to services / dashboards on labs. W... [16:42:16] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2207412 (10Dereckson) There are 404 now, it's probably the time for the cache to expire. [16:47:31] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2207429 (10BBlack) @Milimetric - I guess what I'm missing here is the disconnect between our public termination of analytics.wikimedia.org (on, say, cache_misc) and what "a subfolder on a... [16:54:39] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2207462 (10BBlack) (for services small enough to not need a cluster of their own hardware, I think we do have solutions where we virtualize smaller services on ganeti, too. The above is... [17:02:53] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2207518 (10NahidSultan) >>! In T109331#2207412, @Dereckson wrote: > There are 404 now, it's probably the time f... [17:05:03] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2207527 (10Dereckson) CTRL + SHIFT + R should do the trick in your browser. It's not possible to invalidate a c... [17:14:14] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2207564 (10NahidSultan) >>! In T109331#2207527, @Dereckson wrote: > It's not possible to invalidate a cached fi... [17:37:01] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2207677 (10matmarex) [17:37:32] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2204871 (10matmarex) [19:40:46] 07HTTPS, 10Traffic, 06Operations, 10pywikibot-compat, 10pywikibot-core: pywikibot support for https-only - https://phabricator.wikimedia.org/T102315#2208073 (10Andrew) As I understand it, this ticket is a request for updates to the pywikibot code. I'm going to remove the Operations tag; please re-add wi... [19:46:54] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-IRC-RC-Server, and 2 others: Remove the "HTTPS to HTTP" url filter in the IRC feed - https://phabricator.wikimedia.org/T122933#2208105 (10Andrew) p:05Triage>03Normal [19:48:12] 10Wikimedia-Apache-configuration, 07HHVM: Transition to HHVM broke old links to wiki.phtml - https://phabricator.wikimedia.org/T122629#2208108 (10Andrew) [19:57:24] 10Traffic, 06Operations, 07Beta-Cluster-reproducible: PHP fatal errors causing Varnish to return 503 - "Junk after gzip data" - https://phabricator.wikimedia.org/T125938#2208148 (10Andrew) p:05Triage>03Normal [20:14:53] 10Traffic, 06Discovery, 10Kartotherian, 10Maps, and 2 others: Set up proper edge Varnish caching for maps cluster - https://phabricator.wikimedia.org/T109162#2208223 (10Gehel) [21:05:43] 10Traffic, 06Operations, 13Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2208402 (10Andrew) p:05Triage>03Normal [21:09:44] 07HTTPS, 10Traffic, 06Operations, 06WMF-Communications, 07Security-Other: Server certificate is classified as invalid on government computers - https://phabricator.wikimedia.org/T128182#2208425 (10Andrew) p:05Triage>03Normal [21:51:42] 07HTTPS, 10Traffic, 06Operations: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2208542 (10Dzahn) Is this just about adding https or also about enforcing it? [21:54:55] 07HTTPS, 10Traffic, 06Operations: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2208547 (10Dzahn) and would this be about a certificate on carbon or can it be behind varnish even though it's an APT mirror? [22:47:23] 07HTTPS, 10Traffic, 06Operations: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2198925 (10BBlack) It's about enabling HTTPS with valid certificates for all 3, but not about enforcing it with a redirect (or anything else more advanced). We actually **can't**... [23:01:27] 07HTTPS, 10Traffic, 06Operations: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2208685 (10BBlack) Considering that watchmouse's own status pages, e.g. http://status.cloudmonitor.ca.com/ and http://stations.status.cloudmonitor.ca.com/ don't offer HTTPS at all (connectio... [23:54:17] 07HTTPS, 10Traffic, 06Operations: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2208814 (10BBlack) They might also have the option to simply use a hostname within their domains rather than bothering with another name of ours. e.g. configuring it as `wikimedia.status.as...