[01:10:43] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 3 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#2347865 (10ori) We are definitely sending duplicate purges. I added some debug logging and saw URLs get purged... [01:54:26] 10Traffic, 06Operations, 06Community-Liaisons (Apr-Jun-2016): Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2347953 (10BBlack) @Whatamidoing - would you like recaps/updates of account lists here in this ticket? [07:47:41] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2348241 (10hashar) I have drawn a summary of web services that ends up on gallium. One is on doc.wikimedia.org the three others are on integrati... [09:18:11] spares also rebooted with 4.4.2-3+wmf2 [12:05:08] 10netops, 06Labs, 06Operations: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2348847 (10faidon) Thanks. Testing again production (e.g. upload.wikimedia.org, but any host including bast1001 would do) would also be a useful data poin... [12:06:19] 10netops, 06Labs, 06Operations: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671#2343755 (10faidon) p:05Triage>03Normal [12:20:42] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2348854 (10SindyM3) Lemonbit says: We have successfully installed the certificate for www.wikilovesmonuments.org (alias wikilovesmonuments.org), SSL connectio... [12:28:32] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2348880 (10Akoopal) 05Open>03Resolved a:03Akoopal Per comment from Sindy, certificate is installed. [14:13:34] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2349181 (10Dzahn) This is great but i cant confirm yet it has been installed. wikilovesmonuments.org uses an invalid security certificate. The certificate is... [14:14:03] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2349182 (10Dzahn) 05Resolved>03Open [14:30:29] 10netops, 06Labs, 06Operations, 10Tool-Labs: Someone seems to be running a port scanner in labs - https://phabricator.wikimedia.org/T136829#2349244 (10Peachey88) [14:44:37] 10netops, 06Labs, 06Operations, 10Tool-Labs: Someone seems to be running a port scanner in labs - https://phabricator.wikimedia.org/T136829#2349321 (10Andrew) @Reedy, you're right, I was briefly confusing the src and dst ports. I'll rename [14:45:01] 10netops, 06Labs, 06Operations, 10Tool-Labs: bitninja upset about us running a crawler - https://phabricator.wikimedia.org/T136829#2349322 (10Andrew) [16:47:07] 10Traffic, 06Operations, 06Community-Liaisons (Apr-Jun-2016): Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2349645 (10Whatamidoing-WMF) If that's considered sensitive information, then definitely not. [17:15:20] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2349820 (10Andrew) [17:26:53] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2349206 (10MoritzMuehlenhoff) The report only contains alerts of the kind: "A visitor reached our honeynet and sent... [17:35:51] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2349206 (10scfc) Maybe I'm missing something, but the link (https://bitninja.io/incidentReport.php?details=91e8f633... [17:38:21] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2349206 (10chasemp) It's hard for me to take the BitNinja reports seriously. Previously, it is my understanding, t... [17:55:25] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2349206 (10doctaxon) @Giftpflanze: Die reden wohl von unserem Bot?! [18:04:42] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2350165 (10Giftpflanze) @doctaxon apparently they do, but it seems the claim is unsubstantiated [18:11:40] 10netops, 06Labs, 06Operations, 10Tool-Labs: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2350197 (10MoritzMuehlenhoff) I've replied to them requesting to whitelist the captcha warnings. Will update the Ph... [18:45:39] 07HTTPS, 10Traffic, 10MediaWiki-Email, 07Easy: Links in MediaWiki emails should respect the user's https preference - https://phabricator.wikimedia.org/T41676#2350412 (10BBlack) This is apparently now about MediaWiki, not about Wikimedia emails (as we're all-HTTPS), so I'm removing the Ops/Traffic tags. [18:45:49] 07HTTPS, 10Traffic, 10MediaWiki-Email, 06Operations, 07Easy: Links in MediaWiki emails should respect the user's https preference - https://phabricator.wikimedia.org/T41676#2350417 (10BBlack) [18:49:37] 10Traffic, 06Operations, 06Community-Liaisons (Apr-Jun-2016): Help contact bot owners about the end of HTTP access to the API - https://phabricator.wikimedia.org/T136674#2350457 (10BBlack) Well that's been a question. The raw logs with IP addresses are sensitive. Username lists have been sent to mailing li... [19:02:32] heh I was puzzled at why the sudden shift of tons of percentage from front-hit to front-local for the maps cluster a day ago in https://grafana.wikimedia.org/dashboard/db/varnish-caching ... [19:02:59] it's because of the pybal change. that's the percentage of maps' traffic that comes from pybal healthchecks :) [21:35:57] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, 13Patch-For-Review: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#2351132 (10chasemp) @qgil sort of a [[ https://en.wikipedia.org/wiki/Hail_Mary_pass | hail mary ]] ping as I'm not sure where to raise the alarm on this. It... [22:44:44] bblack: so now we know the healthchecks work :) [22:45:55] I was reading the bloom filters stuff before, it's exciting [22:48:37] 07HTTPS, 10Traffic, 06Operations: Preload STS for wikimedia.org - https://phabricator.wikimedia.org/T132685#2351388 (10BBlack) [22:48:40] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2351389 (10BBlack) [22:48:43] 07HTTPS, 10Traffic, 06Operations: Invalid web certificate on status.wikimedia.org - https://phabricator.wikimedia.org/T123135#2351390 (10BBlack) [22:48:46] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2351385 (10BBlack) 05Open>03Resolved a:03BBlack I've moved the stats.wm.o DNS to wikitech-static, and set up an apache reverse proxy there with a LetsEncrypt cert... [22:49:30] It's alive! https://status.wikimedia.org/ [22:58:50] 07HTTPS, 10Traffic, 06Operations: Preload STS for wikimedia.org - https://phabricator.wikimedia.org/T132685#2351398 (10BBlack) We're now basically in shape to do this. I'd like to wait a few days and see how https://status.wikimedia.org/ works out first. Then we can start running through and setting preloa... [23:18:55] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: status.wikimedia.org has no (valid) HTTPS - https://phabricator.wikimedia.org/T34796#2351571 (10BBlack) (note, above has been edited a few times to correct missing stuff, will keep doing that so this task serves as a good reference) [23:55:21] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2351644 (10Akoopal) intresting. The site fixed is www.wikilovesmonuments.org. without www it seems to just point to the default site