[00:31:48] 10HTTPS, 10Traffic, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.beta.wmflabs.org expired on January 12, 2021. - https://phabricator.wikimedia.org/T271808 (10Krenair) >>! In T271808#6739578, @Vgutierrez wrote: > `root@deployment-cache-upload06:/etc/acmecerts/unified/live# openssl x509 -da... [08:43:11] 10Traffic, 10Phabricator, 10SRE: Excessive queries from vscode-phabricator - https://phabricator.wikimedia.org/T271528 (10Aklapper) Patch has been merged. Is there more to do or can this task be resolved? [10:08:20] 10Traffic, 10Performance-Team, 10SRE: ats-be occasional system CPU usage increase - https://phabricator.wikimedia.org/T265625 (10ema) 05Open→03Resolved a:03ema Erratic CPU usage [[ https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=1&orgId=1&from=1610249262871&to=1610618827835&var... [10:37:00] 10Acme-chief, 10Beta-Cluster-Infrastructure: Issues with acme-chief cert rotation on deployment-prep, 2021-01-12 - https://phabricator.wikimedia.org/T271778 (10AlexisJazz) No relation with T267561? [10:40:04] 10Traffic, 10Phabricator, 10SRE: Excessive queries from vscode-phabricator - https://phabricator.wikimedia.org/T271528 (10jbond) >>! In T271528#6746757, @Aklapper wrote: > Patch has been merged. Is there more to do or can this task be resolved? @Aklapper This task is here for users who may hit the rate limi... [12:56:50] 10Traffic, 10Phabricator, 10SRE: Excessive queries from vscode-phabricator - https://phabricator.wikimedia.org/T271528 (10Aklapper) 05Open→03Resolved :-/ Well, Phab is not a support desk for general questions; it's a task tracking system to plan work. Closing. [13:08:19] 10Traffic, 10SRE, 10ops-eqiad: lvs1016 interface down - https://phabricator.wikimedia.org/T271087 (10BBlack) @Cmjohnson - Please do it at your earliest convenience. It's not in the flow of live traffic and doesn't need any "depool" AFAIK (but it is problematic that we don't have it as a reliable backup opti... [13:27:43] 10Acme-chief, 10Beta-Cluster-Infrastructure: Issues with acme-chief cert rotation on deployment-prep, 2021-01-12 - https://phabricator.wikimedia.org/T271778 (10Krenair) Unlikely [14:58:10] 10Traffic, 10SRE, 10ops-eqiad: lvs1016 interface down - https://phabricator.wikimedia.org/T271087 (10elukey) I added a week of downtime, the alarm popped up again, remember it if the issue gets solved sooner! [15:20:18] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) Adding bblack (traffic) and ayounsi, as we have a tentative date for codfw, proposed by Filippo and Jaime: week of 15th. It may change as persiste... [15:20:31] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) a:05jcrespo→03None [15:22:03] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [15:22:32] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [16:37:35] 10Traffic, 10Analytics, 10SRE: Traffic anomalies: Factor out list of countries into a dedicated Hive table - https://phabricator.wikimedia.org/T272052 (10mforns) [18:11:39] 10Traffic, 10Analytics, 10SRE: Traffic anomalies: Factor out list of countries into a dedicated Hive table - https://phabricator.wikimedia.org/T272052 (10razzi) p:05Triage→03Medium [18:13:53] 10Traffic, 10Analytics, 10Analytics-Kanban, 10SRE: Traffic anomalies: Factor out list of countries into a dedicated Hive table - https://phabricator.wikimedia.org/T272052 (10razzi) p:05Medium→03High [18:32:35] 10Traffic, 10SRE, 10ops-eqiad: lvs1016 interface down - https://phabricator.wikimedia.org/T271087 (10Cmjohnson) 05Open→03Resolved @elukey @bblack swapped both the optics at the switch on a4 and on the server. It appears that the server side optic was the issue, the link is backup xe-4/0/7 up... [19:23:31] traffic folk, I would appreciate a couple more +1s on https://gerrit.wikimedia.org/r/c/operations/puppet/+/655761 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/655762 before I merge them. Thanks! [23:00:59] 10Acme-chief, 10Beta-Cluster-Infrastructure: Issues with acme-chief cert rotation on deployment-prep, 2021-01-12 - https://phabricator.wikimedia.org/T271778 (10AlexisJazz) >>! In T271778#6747588, @Krenair wrote: > Unlikely I was asking because T267006#6624466 (deployment-cache-upload06 is upload.beta.wmflabs....