[06:21:20] hello hello [06:21:43] afaics the Lumen link esams-eqiad is down but not due to a scheduled maintenance (can't find any email for that) [06:22:30] and in netbox I can see that they want either a phone call or a ticket opened in the portal, but I don't have credentials (or don't know where to find them) [06:23:02] the laser rx pw on the eqiad side is not good [06:23:02] Laser receiver power : 0.0003 mW / -35.23 dBm [06:26:20] can anybody double check that I am not crazy/de-caffeinated in ops-maintenance@ and/or noc@ ? [06:31:38] I can't see anything either, so either we're equally crazy/de-caffeinated or they actually didn't send a notice [06:32:08] :D [06:32:09] thanks! [06:32:36] do you know by any chance where we store the credentials to open tickets in the Lumen portal? [07:15:34] elukey: looking [07:15:48] elukey: they're personal credentials :( [07:16:48] XioNoX: ahhh only elite SREs can have them okok [07:16:50] :D [07:18:25] elukey: I wish I could make you think kind of elite [07:18:40] ahahhah [07:19:24] elukey: looks like it's back? [07:19:41] XioNoX: it seems so yes [07:19:58] Is Lumen the new Zayo? [07:20:34] hopefully not :D [07:23:48] seems like it started acting up around 3 hours ago, judging from the alerts in the channel, so that's a long time to be on-again-off-again [10:56:12] I'm stealing the wmf's systemd::timer::job code and want something to run every 10 minutes but I can't find the righ format for onCalendar [11:05:01] RhinosF1: the following should do it [11:05:03] interval => '*-*-* *:00/10:00', [11:07:55] RhinosF1: you may also fine the following usefull [11:07:57] systemd-analyze calendar '*-*-* *:00/10:00' [11:12:52] jbond42: https://github.com/MirahezeBots/puppet/blob/master/modules/puppet/manifests/init.pp gives me Error: Evaluation Error: Error while evaluating a Resource Statement, Systemd::Timer::Job[run-puppet]: [11:12:52] parameter 'interval' variant 0 entry 'interval' expects a match for Variant[Systemd::Timer::Interval = Pattern[/\A(\d+\s*(us|ms|s|sec|m|min|h|hour|d|w|M|y)\s*)+\Z/], Systemd::Timer::DateTime = Pattern[/\A((Mon(,|\.\.)?|Tue(,|\.\.)?|Wed(,|\.\.)?|Thu(,|\.\.)?|Fri(,|\.\.)?|Sat(,|\.\.)?|Sun(,|\.\.)?)*\s){0,1}(\d{4}|\*)-(\d{2}|\*)-(\d{2}|\*)\s\d{2}(\/\d{1,2}){0,1}:\d{2}(\/\d{1,2}){0,1}:\d{2}(\/\d{1,2}){0,1}\Z/]], got '*-*-* *:00/10:00' [11:12:52] parameter 'interval' variant 1 expects an Array value, got Struct (file: /etc/puppet/code/modules/puppet/manifests/init.pp, line: 2) on node 112.node1.net.fosshost.org [11:18:06] RhinosF1: strange i just checked and seems to parse for me https://gist.github.com/b4ldr/c569f829ce576847dbfe7380d5fd581a [11:18:45] also the error messages says interval got a Struct when it clearly gets a string [11:26:12] jbond42: fun! [11:29:37] jbond42: try just copy our modules folder and test on that in case I've messed my copying up [11:34:26] RhinosF1: seems to work https://gist.github.com/b4ldr/c5110491cd23736c4fbd67b784ce87e4 [11:35:06] jbond42: then what is puppet doing [11:35:14] What puppet version you on ? [11:36:00] RhinosF1: get the same with puppet 5.5.22 and 7.3.0 [11:36:14] Ours is 7 something [11:46:37] fyi -operations has some icinga alarms related to authdns config failures [11:46:55] RhinosF1: looks like you are missing https://gerrit.wikimedia.org/r/c/operations/puppet/+/584020 [11:47:07] probably best to just re sync the systemd module [11:58:01] jbond42: resyncing that folder gives [11:58:03] Error: Evaluation Error: Error while evaluating a Resource Statement, Systemd::Timer::Job[run-puppet]: parameter 'interval' expects a value of type Systemd::Timer::Schedule = Struct[{'start' => Systemd::Timer::Start = Enum['OnActiveSec', 'OnBootSec', 'OnCalendar', 'OnStartupSec', 'OnUnitActiveSec', 'OnUnitInactiveSec'], 'interval' => Variant[Systemd::Timer::Interval = Pattern[/\A(\d+\s*(us|ms|s|sec|m|min|h|hour|d|w|M|y)\s*)+\Z/], [11:58:03] Systemd::Timer::Datetime = String]}] or Array[Systemd::Timer::Schedule = Struct[{'start' => Systemd::Timer::Start = Enum['OnActiveSec', 'OnBootSec', 'OnCalendar', 'OnStartupSec', 'OnUnitActiveSec', 'OnUnitInactiveSec'], 'interval' => Variant[Systemd::Timer::Interval = Pattern[/\A(\d+\s*(us|ms|s|sec|m|min|h|hour|d|w|M|y)\s*)+\Z/], Systemd::Timer::Datetime = String]}], 1], got String (file: [11:58:03] /etc/puppet/code/modules/puppet/manifests/init.pp, line: 2) on node 112.node1.net.fosshost.org [12:04:36] fixed that [12:04:44] now this https://www.irccloud.com/pastebin/YEniAwdE/ [12:11:33] RhinosF1: you will need to delete all refrences to $::initsystem see https://gerrit.wikimedia.org/r/c/operations/puppet/+/677558 [12:12:09] https://www.irccloud.com/pastebin/AMUCF834/ [12:12:23] jbond42: I did that and got ^ [12:13:41] i also had to updated stdlib and copy rsyslogd, logrotate and debian mopdules. further had to patch rsyslog ro remove the refrence to base::service_auto_restart [12:16:36] RhinosF1: you should use systemctl cat puppet-run to get a better idea [12:17:18] rhinosf1@112:~$ systemctl cat run-puppet [12:17:18] # Unit run-puppet.service could not be loaded. [12:18:10] RhinosF1: akaik execstart neets to be one command [12:18:45] wikibugs died again? [12:18:49] I will restart [12:21:18] jbond42: how do you find something on path, which/whereis isn't giving anything because there is a puppet-run bash script somewhere. [12:22:20] marostegui: did you do anything to wb2-grrrrit? or just the irc one? [12:22:46] Majavah: I am following: https://www.mediawiki.org/wiki/Wikibugs [12:22:56] all the jobs are running now btw [12:23:05] ah, thanks [12:25:18] looks like it is back now [12:26:46] yep [12:27:03] it only joins #-cloud when starting up, it will join other channels only when needed for the first time [12:29:42] fixed jbond42 [12:42:27] RhinosF1: \o/ [12:43:33] jbond42: you'd be surprised to here how strange our setup is [12:46:51] RhinosF1: :D [12:54:59] jbond42: I'm on a mission to make things as sensible as possible before bullseye comes out