[01:57:24] PROBLEM - Puppet staleness on tools-worker-1005 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [02:15:10] Change on 12www.mediawiki.org a page Talk:Developer access was modified, changed by Danwe link https://www.mediawiki.org/w/index.php?diff=2250599 edit summary: [+357] /* Who can help resetting password on wikitech.wikimedia.org ? */ new section [03:44:45] if a user can not remember their password, and reset password doesn't work, then who should they ask? [04:35:05] PROBLEM - Free space - all mounts on tools-worker-1018 is CRITICAL: CRITICAL: tools.tools-worker-1018.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found)tools.tools-worker-1018.diskspace.root.byte_percentfree (<22.22%) [06:59:32] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2681478 (10yuvipanda) Well, looks like that particular user account still can not log in, and I haven't had time to look into it yet. I'm on vacation for a week now, so maybe I can look at it after? [08:09:31] PROBLEM - Puppet staleness on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [08:22:32] PROBLEM - Free space - all mounts on tools-exec-gift is CRITICAL: CRITICAL: tools.tools-exec-gift.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-gift.diskspace.root.byte_percentfree (<33.33%) [08:37:30] RECOVERY - Free space - all mounts on tools-exec-gift is OK: OK: tools.tools-exec-gift.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [09:10:13] (03PS1) 10Alexandros Kosiaris: Add replication_pass for eqiad maps servers [labs/private] - 10https://gerrit.wikimedia.org/r/313651 [09:13:11] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add replication_pass for eqiad maps servers [labs/private] - 10https://gerrit.wikimedia.org/r/313651 (owner: 10Alexandros Kosiaris) [09:33:12] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:35:32] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:37:14] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:37:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:38:00] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:38:11] PROBLEM - Puppet run on tools-prometheus-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:38:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:38:25] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:39:51] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:40:17] PROBLEM - Puppet run on tools-k8s-master-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:40:25] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:40:25] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:40:37] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:42:17] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:42:29] PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:42:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:43:29] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:44:20] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:44:30] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:45:12] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:45:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:45:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:45:42] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [09:46:00] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:46:12] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:47:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:47:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:47:36] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:47:51] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:48:41] PROBLEM - Puppet run on tools-exec-cyberbot is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:49:13] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:49:15] 06Labs, 10Labs-Infrastructure, 06Operations, 07Puppet: puppet-enc failure does not produce stderr/stdout printed - https://phabricator.wikimedia.org/T147111#2681529 (10hashar) [09:49:17] PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:49:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:51:53] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:52:15] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:52:17] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:52:21] ouch [09:52:29] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:53:58] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:54:18] PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:55:02] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:55:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:55:28] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:55:50] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:55:51] 06Labs, 10Labs-Infrastructure, 06Operations, 07Puppet: puppet-enc failure does not produce stderr/stdout printed - https://phabricator.wikimedia.org/T147111#2681541 (10hashar) ``` root@deployment-puppetmaster:~# /usr/local/bin/puppet-enc deployment-db03.deployment-prep.eqiad.wmflabs classes: ['role::mariad... [09:55:58] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:56:48] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:57:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:57:27] am I here? Apparently I was banned [09:57:34] yuvipanda: yeah [09:57:35] you are [09:58:15] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:58:33] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:58:39] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:59:13] 06Labs, 10Labs-Infrastructure, 06Operations, 07Puppet: puppet-enc failure does not produce stderr/stdout printed - https://phabricator.wikimedia.org/T147111#2681545 (10hashar) The script uses `/etc/ldap.yaml` and is apparently run on the puppet master. It has the servers: - ldap-labs.eqiad.wikimedia.or... [09:59:27] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:00:15] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:00:37] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:00:39] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:01:53] 06Labs, 10Labs-Infrastructure, 06Operations, 07Puppet: puppet-enc failure does not produce stderr/stdout printed - https://phabricator.wikimedia.org/T147111#2681548 (10hashar) [10:02:26] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:02:26] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:02:38] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:03:12] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:03:28] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:03:36] !log tools re-enable puppet on tools-checker-02 [10:03:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [10:04:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:04:24] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:05:14] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:06:53] PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:07:45] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:08:05] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:08:21] yuvipanda: plane's boarding, so i'm getting off laptop [10:08:29] madhuvishy: yeah, np [10:08:39] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:10:05] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:12:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:12:46] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:13:09] RECOVERY - Puppet run on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:13:14] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:18] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:14:32] RECOVERY - Puppet staleness on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [10:15:02] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:15:58] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:16:50] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:16:52] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:17:16] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:21] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:17:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:17:51] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:18:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:25] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:18:39] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:49] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:19:13] (03PS1) 10Alexandros Kosiaris: Revert "Add replication_pass for eqiad maps servers" [labs/private] - 10https://gerrit.wikimedia.org/r/313657 [10:20:19] RECOVERY - Puppet run on tools-k8s-master-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:27] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:03] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:22:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:40] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:22:44] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:23:39] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:24:17] RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:19] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:49] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:25:07] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:25:11] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:29] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:42] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:26:04] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:26:48] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:27:46] RECOVERY - Puppet run on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:24] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:28:40] RECOVERY - Puppet run on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:50] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:29:14] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:29:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:28] PROBLEM - Puppet run on tools-exec-1211 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:32:15] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:19] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [10:33:33] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:34:29] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:35:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:37:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:14] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:16] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [10:40:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [10:40:42] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [10:42:22] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:42:26] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [10:42:36] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [10:43:26] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:44:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [10:44:23] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:13] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:25] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:31] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [10:46:43] 06Labs, 10Labs-Infrastructure, 07LDAP: Make LDAP scripts that use ldap3 do failover rather than round robin - https://phabricator.wikimedia.org/T147112#2681555 (10yuvipanda) [10:46:53] RECOVERY - Puppet run on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:15] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:29] RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:43] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:03] RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:25] RECOVERY - Puppet run on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:27] (03Abandoned) 10Alexandros Kosiaris: Revert "Add replication_pass for eqiad maps servers" [labs/private] - 10https://gerrit.wikimedia.org/r/313657 (owner: 10Alexandros Kosiaris) [10:49:30] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:50:04] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:50:40] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [10:52:44] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:52:54] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:02] RECOVERY - Puppet run on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [10:54:16] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [10:55:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:03] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:03] 06Labs, 10Labs-Infrastructure, 06Operations, 07Puppet: puppet-enc failure does not produce stderr/stdout printed - https://phabricator.wikimedia.org/T147111#2681567 (10hashar) 05Open>03Resolved Got fixed by restarting some services. T147112 is the follow up actionable. [10:56:53] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [10:57:37] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [10:57:49] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:57:50] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:29] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:49] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [10:59:12] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:02] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:18] Change on 12www.mediawiki.org a page Talk:Developer access was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=2250647 edit summary: [+167] /* Who can help resetting password on wikitech.wikimedia.org ? */ re [11:00:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:00] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:14] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:50] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:54] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:30] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:38] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:46] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:26] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:04:18] RECOVERY - Puppet run on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:00] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:02] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:02] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:46] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:50] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [11:07:40] RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:25] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:39] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:39] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:51] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:59] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [11:09:15] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:05] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:15] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:29] RECOVERY - Puppet run on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [12:29:12] yuvipanda: I created a new tool called iabot. It seems to be quite empty. [12:29:48] Other than the cnf file, there's nothing there. shouldn't there be more folders like the public_html folder? [12:31:21] andrewbogott: ^ [12:34:37] I don't think the tool was created properly. Any assistance would be welcome. [12:52:04] There should be a limit to stop shinken-wm spamming like ^ [12:52:28] tom29739: ? [12:53:14] It was spamming for about 3ish hours, from 10am till 1pm [12:53:40] What time zone? I don't see anything. [12:54:27] derp, I have that thing on my /ignore list. [12:54:37] So I can't see it. :p [12:56:07] Yeah, it spams me. [12:56:14] If I'm here then I quiet it. [12:56:39] tom29739: you're an op? [13:01:53] CP678|Laptop_, no [13:02:24] CP678|Laptop_, but wm-bot is, and I have admin permissions on that. [13:02:31] So I use @q [13:02:37] @q shinken-wm [13:02:48] :O [13:02:50] @unq shinken-wm [13:02:55] @q tom29739 [13:03:01] @unq tom29739 [13:03:13] I trust: .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikimedia/Ryan-lane (2admin), .*@wikipedia/.* (2trusted), .*@nightshade.toolserver.org (2trusted), .*@wikimedia/Krinkle (2admin), .*@[Ww]ikimedia/.* (2trusted), .*@wikipedia/Cyberpower678 (2admin), .*@wirenat2\.strw\.leidenuniv\.nl (2trusted), .*@unaffiliated/valhallasw (2trusted), .*@mediawiki/yuvipanda (2admin), .*@wikipedia/Coren (2admin), .*@wikimedia/BDavis-WMF (2admin), .*@wikimedia/Krenair (2admin), .*@wikimedia/mviswanathan-wmf (2admin), [13:03:13] @trusted [13:03:21] You're an admin ^ [13:03:28] I forgot I that existed, and that I had that power. :p [13:03:47] <|L> but if you quiet shinken-vm, better do that with shinken-vm*!*@* [13:03:56] |L, you can't [13:04:06] <|L> otherwise ohter bots may be quiet too [13:04:06] <|L> *other [13:04:07] It only accepts users as arguments [13:04:19] <|L> then this is something to do :P [13:04:22] I'm not an op myself, so I can only do it that way [13:04:46] You are admin and identified by the name .*@wikipedia/Cyberpower678 [13:04:46] @whoami [13:04:51] tom29739: ^ [13:04:55] use that. [13:05:16] You are admin and identified by the name .*@wikipedia/Cyberpower678 [13:05:16] @whoami tom29739 [13:05:22] :| [13:06:16] Oh well. [13:19:58] You are admin and identified by the name .*@wikipedia/tom29739 [13:19:58] @whoami [13:20:03] CP678|Laptop_, ^ [13:20:20] (had to do go away) [13:20:48] CP678|Laptop_, it won't appear in that list because I'm a global admin, and not just this channel. [13:22:15] <|L> tom29739: btw, is there a way to see the global admin list? :O [13:22:25] |L, no [13:22:50] |L, I can tell you it though, there's only something like 4 or 5 global admins/roots [13:22:59] <|L> ah, ok [13:23:03] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.8.0.0 [libirc v. 1.0.3] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [13:23:03] @help [13:24:21] |L, there's petan, jeremyb, thehelpfulone, sDrewth as the original roots [13:24:50] |L, and Matthew_ as another global root [13:24:58] |L, and me as a global admin. [13:25:24] <|L> ah, ok [13:25:30] I don't think I've forgotten anyone. [13:29:17] I don't need to be a global admin. :p [13:41:12] (03CR) 10Jean-Frédéric: "> IS it possible to use the harvesting in the bot development to" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [13:41:18] It comes with a lot of responsibility. [13:52:25] (03CR) 10Jean-Frédéric: Allow to set lang parameter in update_database (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313451 (owner: 10Jean-Frédéric) [14:10:45] tom29739: ping? [14:22:55] Matthew_, oh, was just telling |L who the wm-bot roots/admins were [14:26:30] Ah, OK :) [15:36:28] tom29739: I thought T13 is the second original root? [15:36:28] T13: Plan to migrate everything to Phabricator - https://phabricator.wikimedia.org/T13 [15:36:50] * zhuyifei1999_ meant Technical13 [15:57:20] zhuyifei1999_, yep, he's a root [15:57:42] He's not on the support page though, so I forgot them [17:30:33] PROBLEM - Puppet staleness on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [17:31:29] yuvipanda: I seem to be experiencing a serverside caching problem. Is there any thing I can do in PHP, forcefully bypass the cache when needed. It seems to be messing up my OAuth session. [17:31:44] On tool labs. Particularly the iabot tool. [17:39:35] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/20after4 was modified, changed by 20after4 link https://wikitech.wikimedia.org/w/index.php?diff=874171 edit summary: [17:46:25] huh, I can't select from information_schema.views ? [17:50:51] nope, I'm just stupid [17:50:57] never mind [17:52:37] <|L> twentyafterfour: Sure, that you need to update that request? I don't think so... I mean you got access to full toollabs, so why do you need an request? [18:07:05] |L: For one thing, I wanted to try out the toollabs interface and I'm not in the right security groups [18:07:17] maybe I can approve my own request? not sure about that. [18:07:30] <|L> twentyafterfour: which interface? [18:07:32] twentyafterfour, can't you create new tools? [18:07:41] I've never used toollabs [18:07:52] I have used other labs projects [18:08:12] I get "projectadmin role required in project tools" [18:08:27] What are you trying to do? [18:08:28] oh that's the wrong place [18:08:34] I just wanted to create a tool really [18:09:00] This is the page: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools [18:09:08] Just enter a name for it [18:09:25] It has to be a unix user, so it's limited to that [18:09:37] service group = a tool [18:09:47] ok [18:10:11] Then you can just "become " [18:11:46] cool [18:12:10] I didn't know it was the same thing as a service group. I never actually understood what a service group was for ;) [18:14:11] a 'tool' is just the special tools project name for a service group [18:14:26] some other projects have service groups too [18:14:44] I see [18:15:43] they set up extra users, groups and sudo rules from the group members [18:16:01] the become command in tools runs sudo -iu [18:17:07] I see [18:17:18] makes sense [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:07:15] Change on 12www.mediawiki.org a page Talk:Developer access was modified, changed by Danwe link https://www.mediawiki.org/w/index.php?diff=2250765 edit summary: [+225] /* Who can help resetting password on wikitech.wikimedia.org ? */ [21:26:34] Change on 12www.mediawiki.org a page Talk:Developer access was modified, changed by BDavis (WMF) link https://www.mediawiki.org/w/index.php?diff=2250767 edit summary: [+481] /* Who can help resetting password on wikitech.wikimedia.org ? */ [22:47:19] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [23:04:24] 06Labs, 10MediaWiki-API: I get lags during stats reading through API - https://phabricator.wikimedia.org/T147109#2682057 (10Krenair) I suspect the problem is either going to be in tools or upstream in mono. I don't know what headers etc. it's sending. `strace -cf mono T147109.exe` starts with: ```name=tools-ba... [23:20:52] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170)