[04:52:51] <wikibugs>	 10Traffic, 10SRE, 10serviceops, 10Sustainability (Incident Followup), 10Wikimedia-Incident: The safe service restart script doesn't detect failure when running with poolcounter. - https://phabricator.wikimedia.org/T272262 (10Krinkle)
[10:12:02] <wikibugs>	 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Joe) 05Open→03Resolved a:03Joe Given the original issue is definitely resolved, there is no point in keeping this ta...
[10:23:47] <wikibugs>	 10Traffic, 10SRE: cp_upload @ eqsin cascading failures, February 2021 - https://phabricator.wikimedia.org/T274888 (10MoritzMuehlenhoff) p:05Triage→03High
[10:54:04] <wikibugs>	 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) Adding local dc ops on CC of this ticket- things would have to go really bad to needing him for this test (this should be a relatively boring proc...
[11:34:39] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10User-MoritzMuehlenhoff: Create a generic network performance profile - https://phabricator.wikimedia.org/T274230 (10MoritzMuehlenhoff) p:05Triage→03Medium
[13:26:26] <wikibugs>	 10Traffic, 10SRE: HTTP 502 Error when trying to create new page (500k characters) on Romanian Wikisource - https://phabricator.wikimedia.org/T273623 (10MoritzMuehlenhoff) p:05Triage→03Medium
[13:36:44] <cdanis>	 sigh, I think I didn't bump nuke_limit by enough https://w.wiki/zwe
[13:41:37] <wikibugs>	 10Traffic, 10SRE: validate or revert the new large_objects_cutoff & nule_limit settings on upload@eqsin - https://phabricator.wikimedia.org/T275028 (10CDanis)
[13:56:56] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: validate or revert the new large_objects_cutoff & nuke_limit settings on upload@eqsin - https://phabricator.wikimedia.org/T275028 (10CDanis)
[13:58:02] <cdanis>	 https://etherpad.wikimedia.org/p/Traffic-2021-02-17
[14:02:50] <cdanis>	 vgutierrez: if you're around can I get a quick +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/664824/ ?
[14:09:12] <vgutierrez>	 checking
[14:12:14] <cdanis>	 thanks!
[14:16:48] <cdanis>	 argh... I just realized I'm an idiot
[14:17:07] <cdanis>	 that's a *command line flag*, and I never restarted varnishfes to roll out the previous 1000 version
[14:17:34] <cdanis>	 I'm going to revert to the 1000 value, and then do a slow rolling restart in eqsin
[14:18:17] <vgutierrez>	 I wondered about that yesterday
[14:25:58] <vgutierrez>	 cdanis: remember to proceed slowly with the varnish-fe restart :)
[14:26:11] <cdanis>	 vgutierrez: one every 10 minutes sound good?
[14:26:14] <vgutierrez>	 yep
[14:26:17] <cdanis>	 thanks
[16:48:18] <wikibugs>	 10Traffic, 10SRE: provision more machines for eqsin caches - https://phabricator.wikimedia.org/T275046 (10CDanis)
[18:08:01] <wikibugs>	 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests - https://phabricator.wikimedia.org/T269914 (10bd808) This bug and {T274090} are both semi-innocuous in that neither is currently causing IABot or the wikis to break, bu...
[23:25:47] <wikibugs>	 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests - https://phabricator.wikimedia.org/T269914 (10bd808) >>! In T269914#6835578, @Legoktm wrote: > We saw a bunch of these requests again today. The main problem is that ma...