[11:04:10] 10netops, 10Operations, 10observability, 10User-fgiunchedi: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations - https://phabricator.wikimedia.org/T267018 (10fgiunchedi) +netops for visibility, cc @ayounsi [11:11:15] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) The list of VSM-related issues affecting 5.2.1 according to [[https://github.com/varnishcache/varnish-cache/blob/6.0/doc/changes.rst#fix... [11:56:53] 10netops, 10Operations, 10observability, 10User-fgiunchedi: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations - https://phabricator.wikimedia.org/T267018 (10jbond) 05Open→03Resolved a:03jbond Looks like this is complete, resolving please reopen if i mis... [11:57:20] 10netops, 10Operations, 10fundraising-tech-ops: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802 (10jbond) p:05Triage→03Medium [12:02:22] 10netops, 10Operations: Upgrade Routinator 3000 to 0.8.2 - https://phabricator.wikimedia.org/T269738 (10ayounsi) p:05Triage→03Medium [12:05:04] 10netops, 10Operations, 10observability, 10User-fgiunchedi: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations - https://phabricator.wikimedia.org/T267018 (10ayounsi) 05Resolved→03Open Not everything has been migrated yet, see the full list on https://libr... [12:06:29] 10netops, 10Operations, 10observability, 10User-fgiunchedi: LibreNMS sends its alerts to Alertmanager, resulting in email notifications to network operations - https://phabricator.wikimedia.org/T267018 (10jbond) p:05Triage→03Medium >>! In T267018#6678835, @ayounsi wrote: > Not everything has been migra... [12:13:40] 10Domains, 10Traffic, 10Okapi, 10Operations: Okapi Domains - https://phabricator.wikimedia.org/T269686 (10jbond) p:05Triage→03Medium [14:41:02] looks like all's well after geoip updates! I have submitted a patch to clear out that old hack for RES: https://gerrit.wikimedia.org/r/c/operations/dns/+/647253 [14:45:54] thanks again for your help! (including back in 2018 :P) [16:17:35] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) OK the amount of work needed to get 5.2.1 in a usable state really seems excessive. Let's give a try to 6.0.0, which is the version imme... [18:38:00] 10Domains, 10Traffic, 10Operations: URL to redirect to upcoming Wikipedia Birthday page on wikimediafoundation.org - https://phabricator.wikimedia.org/T264367 (10Dzahn) @hdothiduc @Varnent Done! Added to DNS and https://20.wikipedia.org works now for me. There could be a little delay depending on caches an... [18:41:36] 10Domains, 10Traffic, 10Operations: URL to redirect to upcoming Wikipedia Birthday page on wikimediafoundation.org - https://phabricator.wikimedia.org/T264367 (10Dzahn) If you can confirm things are working for you then it's up to you if we close this ticket now or after the actual birthday page has been cre... [20:47:24] Is cloud VPS 'traffic-dnsbox.traffic.eqiad.wmflabs [20:47:47] used and by this team? [20:48:49] I have some puppet refactoring that touches an NTP class used only on that.. but puppet is pre-broken for other reasons. And I would have liked to confirm noop. [20:49:07] and since puppet is broken since quite some time there the wmcs team would ping for other reasons [20:49:47] after saying this I notice the reason is not "puppet error" or "puppet disabled" ..no . it's "no space left on device" oops [20:53:24] I will gzip a very large daemon.log.1 in an attempt to hotfix it [20:53:40] logging in project SAL in -cloud [20:54:52] mutante: what NTP clas is used only on that? [20:54:54] *class [20:55:14] but yeah I'd say it's fair to assume that's a derelict VPS instance that could/should be shut down [20:57:07] bblack: [20:57:07] Puppet Class: role::dnsbox [20:57:23] and that uses profile::ntp [20:58:13] but the way Moritz/Jbond commented there are plans to use it. it has a $use_chrony option [20:58:43] we do use role::dnsbox in production [20:59:04] that's what I'm confused about, maybe I misunderstood you're earlier statement [20:59:15] ugh "your" [21:00:13] https://gerrit.wikimedia.org/r/c/operations/puppet/+/645206/4/modules/profile/manifests/ntp.pp [21:00:39] https://gerrit.wikimedia.org/r/c/operations/puppet/+/645206/ [21:00:58] So I looked up where it's used with https://openstack-browser.toolforge.org/puppetclass/role::dnsbox and there was a comment it was NOT used in production. [21:01:31] what comment? [21:01:42] oh yea, and what made me believe that originally is I put the class name in puppet compiler and that found no hosts [21:02:02] anyways, in case we're just getting hung up on language: all the dns* boxes in production (e.g. dns3001.esams.wmnet) use that role. [21:02:02] on PS4: "Looks good (and yes, it's currently not yet used in production)" [21:02:15] but they don't $use_chrony [21:02:27] (that was an experiment that hasn't ever gotten off the ground) [21:03:35] ACK, i see how there is role(dnsbox) on prod machines in site.pp and that uses profile::ntp as well [21:04:44] trying the compiler run on "C:profile::ntp" again [21:06:28] and.. it works. not sure what we did not see there before or happened [21:06:32] dns3001 - https://puppet-compiler.wmflabs.org/compiler1001/27062/dns3001.wikimedia.org/index.html [21:06:50] it's just that the use_chrony thing moves to parameters but stays default false [21:07:39] regarding the cloud VPS instance, i just deleted one of the logs, the older one and tried to run puppet agent once, it has an actual puppet problem now where it can't find something in Hiera. but I won't worry about it then. [22:20:24] I am shutting down that "traffic-dnsbox" instance. But I am not going so far to delete it completely. [22:21:42] after mentioning it in cloud channel and getting a response to delete it