[06:52:15] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:59:18] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:32:15] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:20] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:53] PROBLEM - Puppet staleness on tools-k8s-master-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [14:13:52] Can somebody help me? I recived a mail that puppet fails at my instance xenon.rcm.eqiad.wmflabs [14:19:44] Luke_: check /var/log/puppet.log on that host [14:20:40] valhallasw`vecto: Ignoring stale puppet agent lock for pid [14:20:40] ^[[mNotice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)^[[0m [14:20:55] it's a bunch of lines like that [14:21:04] do you know, what to do in that case? reboot? [14:21:31] Luke_: check if there is a puppet process currently running; if not, remove that lock file [14:25:57] valhallasw`vecto, all puppet processes and the lock file are now gone. what to do then? just wait, or need to use a command? [14:26:31] Luke_: you can either manually run puppet, or wait for cron to automatically run it [14:27:13] ok, thx :) [19:13:32] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2664654 (10Framawiki) Ping @yuvipanda :) [19:15:13] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2664657 (10yuvipanda) Hello! I unfortunately won't have time to look into this for at least two more weeks (all of ops are travelling to our offsite). I'll check back with @Tgr and debug wh... [19:25:31] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2664700 (10Framawiki) No problem, I understand. I have a shell acces on tool labs too, so it is not urgent. And yes i have checked tonight, my second account work on paws, so i can run my t... [19:48:08] is it possible to have the LDAP (labs) account removed? [21:33:15] Looks like labs is having issues connecting to freenode [21:39:05] Ive got several jobs not connecting to irc.freenode.net [21:40:08] Betacommand hi, freenode is currently having a denial attack [21:40:20] ah ok [21:40:35] Betacommand Please see https://twitter.com/freenodestaff/status/779759172518866944 [21:40:52] So you will keep get disconnected and reconnected. Keeps happening to me tonight [22:48:42] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [23:10:30] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [23:16:12] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218)