[07:49:26] ema: about your comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/469686/1/manifests/site.pp#2212 [07:49:54] I remember that new nodes in LVS are depooled by default, isn't that the case? [07:54:30] 10Traffic, 10DNS, 10GitHub-Mirrors, 10Operations, and 2 others: Github: add verified domain - https://phabricator.wikimedia.org/T207364 (10jijiki) p:05Triage>03Normal [08:46:48] gehel: ah, you might be right actually! [08:47:25] anyways, I'd disable puppet on both LVSs before merging just to err on the side of caution :) [08:47:33] in any case, not much of an issue here, the different roles are identical except for the LVS cluster [08:47:56] ok, I'll do that [08:48:29] I'm going to wait Monday to merge them, unless the situation degrades [08:50:41] sounds good! [08:51:13] I am still curious why this machine misbehaves [10:09:20] dear traffic, what priority should I give to this: "Determine cause of upload.wikimedia.org requests routed to text-lb (404 Not Found)" [10:09:26] T207340 [10:09:27] T207340: Determine cause of upload.wikimedia.org requests routed to text-lb (404 Not Found) - https://phabricator.wikimedia.org/T207340 [10:12:36] jijiki: "Normal" seems appropriate [10:12:44] tx [10:20:35] 10Certcentral, 10Operations, 10monitoring: Create icinga checks for certcentral - https://phabricator.wikimedia.org/T207294 (10jijiki) p:05Triage>03Normal [10:21:00] tx jijiki <3 [10:22:13] 10Certcentral, 10Icinga, 10Operations, 10monitoring: Create icinga checks for certcentral - https://phabricator.wikimedia.org/T207294 (10jijiki) [10:22:58] 10Traffic, 10Operations, 10Performance-Team: Investigate 200-300ms increase in responseStart.p75 - https://phabricator.wikimedia.org/T207315 (10jijiki) p:05Triage>03Normal [10:23:33] hehe [10:31:03] morning vgutierrez [10:36:19] vgutierrez, I'm beginning to wonder if we should have certcentral write some file containing data about all the current certs and their status [10:41:22] hmmm yeah, I thought it too [10:41:42] but I don't wanna go down that road honestly [10:44:32] so as long as we don't restart blindly the certcentral service, we should be OK [10:44:50] 99.9% of the time, certcentral should be idling [10:54:36] 10Certcentral, 10Patch-For-Review: Take into account LE rate limits on sensitive operations - https://phabricator.wikimedia.org/T207927 (10Vgutierrez) [13:59:15] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/setup cr2-eqord - https://phabricator.wikimedia.org/T204170 (10Papaul) [14:06:52] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/setup cr2-eqord - https://phabricator.wikimedia.org/T204170 (10Papaul) [14:07:29] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/setup cr2-eqord - https://phabricator.wikimedia.org/T204170 (10Papaul) [14:08:27] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/setup cr2-eqord - https://phabricator.wikimedia.org/T204170 (10Papaul) 05Open>03Resolved [16:06:48] hi :) [16:27:22] hello [16:35:46] hi [16:56:39] So ferm merged my pull request [16:59:31] 10Traffic, 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, and 3 others: Ferm's upstream Net::DNS Perl library bad handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) Ferm merged my pull request... [17:07:38] 10Traffic, 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, and 3 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) [17:44:57] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Ping @nuria too [18:42:51] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Nuria) > This is a bit of a busy week for everyone and especially the security team, but we're going to sync up next week... [19:02:59] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, and 3 others: Remove *.cz domains from WMF's infrastructure - https://phabricator.wikimedia.org/T206923 (10Dzahn) @Urbanecm fyi, i think this one domain is different from the others: wikizdroje.cz has address 198.35.26.96 (that's WMF) all... [19:43:10] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, and 4 others: Remove *.cz domains from WMF's infrastructure - https://phabricator.wikimedia.org/T206923 (10Urbanecm) @Dzahn: Thank you. I simply forgot to update NSSET to the new one. https://www.nic.cz/whois/domain/wikizdroje.cz/ says NSSET... [19:55:10] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/setup cr2-eqord - https://phabricator.wikimedia.org/T204170 (10ayounsi) 05Resolved>03Open Seems like we have a duplicate in Netbox: https://netbox.wikimedia.org/dcim/devices/201/ and https://netbox.wikimedia.org/dcim/devices/1954/ The 2nd o... [20:09:07] 10netops, 10Operations, 10ops-eqiad: Fix missing PDU's for row C eqiad in netbox - https://phabricator.wikimedia.org/T208091 (10Cmjohnson)