[06:32:36] 10Traffic, 10Commons, 10Operations, 10Thumbor: PNG thumbnail gives a 429 error - https://phabricator.wikimedia.org/T169678#3405854 (10Gilles) The rate limits in Thumbor are as close to the old Mediawiki ones as they can be, but there are slight differences. Since this request returned nginx as the server,... [07:12:41] 10netops, 10Operations: zombie rstp configuration - https://phabricator.wikimedia.org/T169637#3405974 (10ayounsi) 05Open>03Resolved Cleaned up. [14:30:25] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3407542 (10Papaul) p:05Triage>03Normal [16:13:09] 10Traffic, 10Operations, 10Pybal: pybal should automatically reconnect to etcd - https://phabricator.wikimedia.org/T169765#3407910 (10ema) [16:13:42] 10Traffic, 10Operations, 10Pybal: pybal should automatically reconnect to etcd - https://phabricator.wikimedia.org/T169765#3407940 (10ema) p:05Triage>03High [16:15:12] 10Traffic, 10Operations, 10Pybal: pybal should automatically reconnect to etcd - https://phabricator.wikimedia.org/T169765#3407910 (10ema) [16:19:38] 10Traffic, 10Operations, 10Pybal: pybal should automatically reconnect to etcd - https://phabricator.wikimedia.org/T169765#3407975 (10ema) As @Joe suggested some days ago, we might want to rewrite pybal's etcd code using https://treq.readthedocs.io/. [16:47:34] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban): Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3408170 (10mmodell) [16:47:52] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban): Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3373930 (10mmodell) Need to merge https://gerrit.wikimedia.org/r/#/c/355869/ [17:48:15] bblack ema: howdy! mikhail from discovery/reading analytics here. quick question! :) do we maintain a blacklist of abusive ip addresses? for example, there's an ip address out of russia that makes almost 900k search api calls per day without a user agent (which is against our ToS iirc) and idk if we even have a blacklist they could be added to. in fact, top 13 users of search API (each making 400-900k calls per day) don't provide a user [17:48:15] agent [19:19:36] we used to have an extensive IP blacklist but it got ditched when we migrated to Varnish [19:49:28] Do we not still block requests with empty UAs? [20:31:17] The UAs show up as "-" in Hadoop/Hive, but I don't know if that's because the user_agent field in webrequest table can't be NULL or if that's just what everyone uses. [20:31:59] * RainbowSprinkles changes his UA to - just to be difficult :p [20:32:54] We also get a lot of requests where the UA is the name of the library/API wrapper, rather than their specific application, which also technically goes against our ToS (https://meta.wikimedia.org/wiki/User-Agent_policy) [20:35:13] Nobody ever reads the TOS ;-) [20:37:26] MaxSem: that's a shame :\ [20:38:33] we've also started to use a rate limiter and I know the empty UA was part of the discussion, but not sure exactly what is in place right now [20:39:15] Feb. 2010! I remember that thread :) [20:39:22] * RainbowSprinkles walks down memory lane for a moment [20:40:02] lol, I was referring to last month stuff :-P [20:40:35] * RainbowSprinkles mutters something about a wiki-lawn and youngins [20:45:41] I think blocking requests without a UA is a must, however any public discussion about that will die in a lot of moaning and accusations. that's why we should just do it' [20:47:42] 10Traffic, 10ArchCom-RfC, 10Operations, 10Services (designing): Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting - https://phabricator.wikimedia.org/T167906#3409286 (10GWicke) This RFC is scheduled for today's IRC meeting, at 2pm SF ti... [20:49:22] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#How_can_I_opt_out_of_Wikipedia.27s_HTTPS_encryption.3F [20:50:21] MaxSem: Well, that's what happened in Feb 2010 ;-) [20:51:19] Huh, I'm actually surprised a G4 could run 10.4.x [20:51:24] #til [21:01:58] bblack: the concurrency limiting RFC (https://phabricator.wikimedia.org/T167906) will be discussed now in #wikimedia-office [21:42:21] 10Traffic, 10Discovery, 10Maps, 10Operations, and 2 others: What is a reasonable per-IP ratelimit for maps - https://phabricator.wikimedia.org/T169175#3409504 (10Gehel) [23:56:44] 10HTTPS, 10Traffic, 10MW-1.30-release-notes, 10Operations, and 2 others: Enable HTTPS for swift clients - https://phabricator.wikimedia.org/T160616#3410075 (10aaron) >>! In T160616#3404157, @fgiunchedi wrote: > @aaron yeah it will need some love, in the meantime I've patched in https support so swiftrepl w... [23:56:52] 10HTTPS, 10Traffic, 10MW-1.30-release-notes, 10Operations, and 2 others: Enable HTTPS for swift clients - https://phabricator.wikimedia.org/T160616#3410076 (10aaron) 05Open>03Resolved a:03aaron