[12:43:48] 10Traffic, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review, 05Security: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2474897 (10elukey) p:05Triage>03High [13:03:03] 10netops, 06Labs, 06Operations: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2474946 (10faidon) 05Open>03Resolved a:03faidon I'm resolving this, as this was primarily a task for the intermittent bandwidth issue. @yuvipanda, f... [13:16:45] 10Traffic, 06Operations, 06Community-Liaisons (Jul-Sep-2016), 13Patch-For-Review: Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2474997 (10MarcoAurelio) @Racso is the owner of BOTzilla. Is that bot still using insecure HTTP? [14:14:48] 10Traffic, 06Operations, 06Community-Liaisons (Jul-Sep-2016), 13Patch-For-Review: Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2475190 (10Racso) Hello. I'm the owner of BOTzilla. As far as I know, the bot is currently inactive. Is it actually d... [14:42:51] 10netops, 06Operations, 10ops-eqiad: cr1/cr2-eqiad: install new SCBs and linecards - https://phabricator.wikimedia.org/T140764#2475397 (10faidon) [14:45:25] 10netops, 06Operations, 10ops-eqiad: Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones - https://phabricator.wikimedia.org/T140765#2475419 (10faidon) [14:45:45] 10Traffic, 06Operations, 06Community-Liaisons (Jul-Sep-2016), 13Patch-For-Review: Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2475434 (10BBlack) The final patch to block insecure HTTP is going out a few hours from now, so if the bot is inactiv... [14:49:36] 10netops, 06Operations, 10ops-eqiad: cr1/cr2-eqiad: install new SCBs and linecards - https://phabricator.wikimedia.org/T140764#2475487 (10faidon) [14:50:25] 10netops, 06Operations, 10ops-eqiad: Upgrade cr1/cr2-eqiad JunOS - https://phabricator.wikimedia.org/T140770#2475516 (10faidon) [15:37:44] 10Traffic, 10Varnish, 06Discovery, 06Maps, and 2 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2475733 (10Gehel) [15:45:47] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 2 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2475770 (10demon) [16:07:28] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 2 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2475862 (10BBlack) [16:40:33] 10netops, 06Operations, 10ops-eqiad: cr1/cr2-eqiad: install new SCBs and linecards - https://phabricator.wikimedia.org/T140764#2476006 (10Cmjohnson) [16:40:35] 10netops, 06Operations, 10ops-eqiad: Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones - https://phabricator.wikimedia.org/T140765#2476003 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson Replaced all 4 PEM's in each mx-480. Turns out we already have high capacity fans in each. We now have 2 spa... [17:07:49] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 4 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2471564 (10cscott) This patch might have broken OCG; see T140789. [17:58:30] 10netops, 06Operations, 10ops-eqiad: Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones - https://phabricator.wikimedia.org/T140765#2476399 (10faidon) Sidenote: on cr1-eqiad, we were bitten again by the same issue as T89999. Like before, I ran `restart chassis-control immediately` which resolved the i... [18:14:02] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 4 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2476469 (10demon) >>! In T140658#2476106, @cscott wrote: > This patch might have broken OCG; see T140789. Yep, th... [18:17:55] 10Traffic, 10Analytics, 10MediaWiki-extensions-CentralNotice, 06Operations: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2476507 (10AndyRussG) @BBlack What are the advantages to using a regex rather than an explicit list? [[ https://www.mediawiki.org/... [18:46:43] 10Traffic, 10Analytics, 10MediaWiki-extensions-CentralNotice, 06Operations: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2476603 (10BBlack) It would just be simpler, but we can do a list. The data we have in Cookies_to_remove is just from Ori's 20-mi... [19:13:00] 10Traffic, 10Analytics, 10MediaWiki-extensions-CentralNotice, 06Operations: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2476731 (10AndyRussG) >>! In T132374#2476603, @BBlack wrote: > It would just be simpler, but we can do a list. Fantastic, thx!!... [19:34:09] 10Wikimedia-Apache-configuration, 06Operations, 07HHVM, 07Wikimedia-log-errors: Fix Apache proxy_fcgi error "Invalid argument: AH01075: Error dispatching request to" (Causing HTTP 503) - https://phabricator.wikimedia.org/T73487#2476874 (10hashar) Awesome! As a side note, the logstash syslog filter discard... [22:26:54] 10Traffic, 06Operations, 06Community-Liaisons (Jul-Sep-2016), 13Patch-For-Review: Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2477796 (10BBlack) 05Open>03Resolved [22:26:56] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, and 2 others: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#2477797 (10BBlack) [22:27:04] 07HTTPS, 10Traffic, 06Operations, 07Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2477800 (10BBlack) [22:27:09] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, and 2 others: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#1490448 (10BBlack) 05Open>03Resolved [22:29:50] 07HTTPS, 10Traffic, 06Operations, 07Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2477809 (10BBlack)