[04:26:55] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892 (10Papaul) 03NEW
[04:28:21] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11330353 (10Papaul)
[04:28:22] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: switch refresh - https://phabricator.wikimedia.org/T408510#11330355 (10Papaul)
[04:34:00] <jinxer-wm>	 FIRING: PurgedHighEventLag: High event process lag with purged on cp5029:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5029 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[04:39:00] <jinxer-wm>	 FIRING: [13x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[04:44:00] <jinxer-wm>	 FIRING: [16x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[05:04:00] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11330375 (10Papaul) p:05Triage→03Medium
[05:39:00] <jinxer-wm>	 FIRING: [17x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[05:44:00] <jinxer-wm>	 FIRING: [16x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[06:24:00] <jinxer-wm>	 FIRING: [17x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[06:29:00] <jinxer-wm>	 FIRING: [20x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[06:34:00] <jinxer-wm>	 FIRING: [25x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[06:39:00] <jinxer-wm>	 FIRING: [27x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[06:44:00] <jinxer-wm>	 RESOLVED: [20x] PurgedHighEventLag: High event process lag with purged on cp5020:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts  - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag
[09:00:42] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11330727 (10cmooney) So in general we have tried to keep the subnetting of our IPv4 /24 consistent at POPs, following the template first set in drmrs (and now...
[10:06:42] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11330872 (10MatthewVernon)
[10:07:14] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11330875 (10MatthewVernon) Updated in the light of review from Android and iOS folks - only change to our list of sizes is the addition o...
[10:50:52] <vgutierrez>	 topranks: any idea why we get p.aged for link saturation in codfw but not in eqsin?
[10:51:08] <vgutierrez>	 https://grafana.wikimedia.org/goto/gKMJEegDR?orgId=1
[10:51:47] <topranks>	 oh wow you got paged for it??  that's great nice 
[10:52:08] <topranks>	 well, obviously it's not good, but I thouhgt the alerting was broken :) 
[10:52:30] <topranks>	 em this is the only "sub rated" circuit we have in the network 
[10:52:49] <topranks>	 I suspect because the shaper is only getting activated on the codfw side it's not firing for the other end 
[10:53:06] <topranks>	 or more to the point the shaper is only on outbound, so the inbound isn't triggered in eqsin 
[10:53:48] <topranks>	 this circuit is going to 10G soon, the order has been placed, so unless it's a big problem we can probably leave 
[10:54:10] <topranks>	 once it's at line-speed both sides the regular alerting will work both ways 
[10:55:02] <vgutierrez>	 yup.. it woke me up at 5am on a Saturday dunno if I'd consider that great :)
[12:28:49] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11331121 (10Sfaci) a:03Sfaci
[12:38:41] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11331144 (10Sfaci) > Remove the limit  I guess that, instead of removing completely the limit, we...
[13:54:37] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11331285 (10Sfaci)
[16:15:48] <wikibugs>	 06Traffic: Global block exception for AddDesc app - https://phabricator.wikimedia.org/T407706#11331674 (10CDanis)
[16:24:53] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11331697 (10Sfaci) Based on our slack conversation:    > @Sfaci  > I was wondering if we could jus...
[16:55:50] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11331739 (10Sfaci)
[16:59:26] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11331773 (10MatthewVernon) A further complication - some wikis (I've found at least fr and de) add a lang{fr,de,...} prefix to the thumb...
[19:44:27] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11332185 (10Sfaci) @ssingh I have a couple of questions about this:  - How long should an experime...
[20:49:04] <wikibugs>	 06Traffic, 10DNS: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11332447 (10SCampos-WMF) Hi @BCornwall, I’ve received a request to update the destination URL of donate.wikipedia25.org and donate.wikipedia25.com.  The Fun...
[21:36:12] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11332556 (10AntiCompositeNumber) The regex in {https://phabricator.wikimedia.org/diffusion/THMBREXT/browse/master/wikimedia_thumbor/handl...