[00:16:36] 10Traffic, 10MediaWiki-Interface, 6Operations: Purge pages cached with mobile editlinks - https://phabricator.wikimedia.org/T125841#2037339 (10BBlack) Probably should've put this here instead of in the parent ticket: ``` I've executed bans for date matching anything up through (and including) Feb 5th now, w... [00:39:54] 10Traffic, 6Operations, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#2037466 (10BBlack) 5stalled>3Resolved resolving for now, unless something new pops up [00:56:24] 10Traffic, 6Operations, 6Performance-Team, 5Patch-For-Review: Disable SPDY on cache_text for a week - https://phabricator.wikimedia.org/T125979#2037572 (10BBlack) 5Open>3Resolved This experiment is done [00:56:52] 10Traffic, 6Operations, 10ops-ulsfo, 5Patch-For-Review: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2037576 (10BBlack) 5Open>3Resolved [00:57:38] 10Traffic, 10MediaWiki-Interface, 6Operations: Purge pages cached with mobile editlinks - https://phabricator.wikimedia.org/T125841#2037579 (10BBlack) 5Open>3Resolved a:3BBlack [06:50:45] hi [09:46:38] 7Varnish: hitting 502 errors on en.wikipedia.org - https://phabricator.wikimedia.org/T127227#2038373 (10fbstj) Apologies for not getting back to you sooner.... I've just spent some time random jumping on en-wp and didn't get any 502's but??? (I suspect this can be closed) [09:54:17] 7Varnish: hitting 502 errors on en.wikipedia.org - https://phabricator.wikimedia.org/T127227#2038399 (10Aklapper) 5Open>3declined a:3Aklapper [10:28:18] 10Traffic, 6Operations: compressed http responses without content-length not cached by varnish - https://phabricator.wikimedia.org/T124195#2038487 (10fgiunchedi) p:5Triage>3Normal still pending an audit of what varnish backends might be affected, particularly apache [11:15:09] 10netops, 6Operations: mr1-ulsfo booted from backup JunOS - https://phabricator.wikimedia.org/T127295#2038580 (10faidon) [11:29:15] 10netops, 6Operations: mr1-ulsfo booted from backup JunOS - https://phabricator.wikimedia.org/T127295#2038580 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVL0JBqh-0X0Il_jxsnQ} [2016-02-18T11:29:12Z] mr1-ulsfo: "request system snapshot media internal sli... [11:36:26] 10netops, 6Operations: mr1-ulsfo booted from backup JunOS - https://phabricator.wikimedia.org/T127295#2038687 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVL0Kqi2W8txF7J0uUA4} [2016-02-18T11:36:22Z] upgrading mr1-ulsfo to its pre-recovery version and re... [12:10:27] 10netops, 6Operations: mr1-ulsfo booted from backup JunOS - https://phabricator.wikimedia.org/T127295#2038851 (10faidon) 5Open>3Resolved I did the following: - `request system snapshot media internal slice alternate` to reformat the primary partition in case it was corrupted - `request system reboot`, whic... [12:18:10] 10Traffic, 6Operations, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#1502780 (10hashar) Might cause T127294. Calls to https://integration.wikimedia.org/ci/api/json ends up being corrupted somehow and that started happening when commits for t... [12:30:05] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038934 (10hashar) I noticed that after some X time, the request pass just fine but subsequent ones all fail. I gave it a try... [12:30:17] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038943 (10hashar) [12:30:51] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2038534 (10hashar) [13:03:46] 10Traffic, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, and 6 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#2039160 (10Amire80) Is there anything left to do in ContentTranslation or cxserver... [13:41:00] 10Traffic, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, and 6 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#2039294 (10BBlack) >>! In T110474#2039160, @Amire80 wrote: > Is there anything lef... [13:50:19] 10Traffic, 6Operations, 10Parsoid, 10VisualEditor, and 2 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#1578478 (10Amire80) [13:51:00] 10Traffic, 6Operations, 10Parsoid, 10VisualEditor, and 2 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#2039324 (10Amire80) >>! In T110474#2039294, @BBlack wrote: >>>! In T110474#2039160, @Amire80 wrote: >> Is there anything left to do in ContentTranslat... [14:41:44] 10Traffic, 6Operations, 10Pybal: pybal health checks are ipv4 even for ipv6 vips - https://phabricator.wikimedia.org/T82747#2039471 (10mark) This was added recently in https://gerrit.wikimedia.org/r/#/c/267008/ [14:43:24] 10Traffic, 6Operations, 10Pybal: pybal health checks are ipv4 even for ipv6 vips - https://phabricator.wikimedia.org/T82747#2039473 (10mark) 5Open>3Resolved a:3mark And according to @bblack this is already deployed, so resolving. :) [14:46:12] 10Traffic, 6Operations, 10Pybal: pybal health checks are ipv4 even for ipv6 vips - https://phabricator.wikimedia.org/T82747#2039498 (10BBlack) 5Resolved>3Open No, it's not :P [18:05:38] 10Traffic, 6Operations: Update prod custom varnish package for upstream 3.0.7 + deploy - https://phabricator.wikimedia.org/T96846#2040513 (10BBlack) Note T127294 was fixed by disabling `do_gzip` on the cache_misc cluster. I suspect that there was a bad interaction there with streamed pass-mode response from t... [18:06:22] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2038534 (10BBlack) 5Open>3Resolved Fixed by disabling do_gzip on the misc... [18:17:42] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2040636 (10hashar) Nodepool is all happy reaching out to Jenkins API since do_... [18:21:37] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2040685 (10BBlack) For further followup, see also: https://phabricator.wikimed... [18:52:36] 7HTTPS, 10Wikimedia-Fundraising, 7Blocked-on-Fundraising-Tech: links.email.donate.wikimedia.org should offer HTTPS - https://phabricator.wikimedia.org/T74514#2040989 (10CCogdill_WMF) 5declined>3Resolved Reopening (and resolving) this task to say Fundraising Tech has successfully set up email click tracki...