[00:02:47] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jfhutson was modified, changed by Jfhutson link https://wikitech.wikimedia.org/w/index.php?diff=840839 edit summary: [00:22:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jfhutson was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=840919 edit summary: [00:34:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Edgars2007 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=840952 edit summary: [01:26:49] 06Labs, 10wikitech.wikimedia.org: Some versions of an image not rendering at all at wikitech - https://phabricator.wikimedia.org/T145811#2642559 (10AlexMonk-WMF) I added some `var_dump`s on silver, tracked down the command it was trying to run, and ran it myself: ```lang=bash www-data@silver:/srv/mediawiki/php... [01:32:10] 06Labs, 10wikitech.wikimedia.org: Some versions of an image not rendering at all at wikitech - https://phabricator.wikimedia.org/T145811#2642572 (10AlexMonk-WMF) Seems https://gerrit.wikimedia.org/r/#/c/303553/4 didn't take effect on silver [01:37:47] 06Labs, 10wikitech.wikimedia.org: Some versions of an image not rendering at all at wikitech - https://phabricator.wikimedia.org/T145811#2642576 (10AlexMonk-WMF) I think https://gerrit.wikimedia.org/r/#/c/293328/ would've caused that to happen - silver has mediawiki::multimedia through openstack::openstack_man... [07:45:51] PROBLEM - Puppet staleness on tools-k8s-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [07:51:26] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2642951 (10elukey) @AlexMonk-WMF and now we have this one :D ``` keystoneclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401) (... [08:07:57] 06Labs, 06Operations: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2642953 (10elukey) [08:42:08] 06Labs, 10wikitech.wikimedia.org: Some versions of an image not rendering at all at wikitech - https://phabricator.wikimedia.org/T145811#2642984 (10MoritzMuehlenhoff) I plan to move mediawiki-firejail-convert to use the generic mediawiki-converters.profile instead of mediawiki-imagemagick.profile, that'll fix... [09:40:26] !log tools.heritage Added ka.wikipedia to pywikibot user_config [09:40:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [10:37:38] 06Labs, 06Operations: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2643240 (10AlexMonk-WMF) >>! In T132422#2642951, @elukey wrote: > @AlexMonk-WMF and now we have this one :D > > ``` > keystoneclient.exceptions.Unauthor... [12:37:46] I have a database in my tools service group. I want this db to be publicly readable (possible to use in quarry). Is it possible? [12:37:51] the db is s51100__wp10_p [12:42:21] It already is publicly readable Amir1 [12:42:41] any user on tools with db credentials in ~/.my.cnf should be able to do "mysql -h tools.labsdb s51100__wp10_p" [12:43:41] don't think it's possible to expose these to quarry, it uses s1.labsdb [12:44:38] Krenair: okay, thanks [12:44:52] you could try I guess [12:45:46] I tried already [12:45:55] it didn't work [12:48:36] yuvipanda: tell me when you're around [13:04:47] !log tools.heritage Deactivate categorisation for ('it', 'it') as some Wikimedia Commons users are unhappy with it. [13:04:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [13:07:32] (03CR) 10Jean-Frédéric: [C: 032] Correctly indent one bit in api/FormatCsv [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310142 (owner: 10Lokal Profil) [13:08:23] (03Merged) 10jenkins-bot: Correctly indent one bit in api/FormatCsv [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310142 (owner: 10Lokal Profil) [13:44:22] legoktm: it should be possible to add that to /versions/. File a task (tool-labs-tools-other) and I'll poke at it or take a shot yourself. Right now it just uses the dbgroup files so it would need to also look at that actual wikiversions.json file to find things that are manual changed. [13:58:16] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:08:14] 06Labs, 10Labs-Infrastructure: New instance first puppet run is broken - https://phabricator.wikimedia.org/T144330#2596502 (10AlexMonk-WMF) Didn't this get fixed in https://gerrit.wikimedia.org/r/#/c/307671/1 ? [14:15:23] 06Labs, 10Horizon: Horizon loses credentials every day - https://phabricator.wikimedia.org/T145703#2638694 (10AlexMonk-WMF) From `modules/openstack/templates/liberty/horizon/local_settings.py.erb`: ```# SESSION_TIMEOUT is in seconds and defaults to 1800. Change to one day. # Note that due to # https://bugs... [15:13:38] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:13:40] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:13:50] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:14:00] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:14:29] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:14:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:14:51] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:14:52] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:15:17] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:15:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:15:52] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:17:08] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:17:12] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:17:12] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:17:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:17:34] PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:00] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:18:02] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:18:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:18:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:14] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:18:22] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:18:36] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:18:40] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:18:42] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:18:44] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:18:48] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:19:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:19:28] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:19:30] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:20:03] PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:20:17] PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:20:25] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:20:39] PROBLEM - Puppet run on tools-exec-cyberbot is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:20:59] PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:21:13] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:22:19] PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:22:19] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:22:47] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:22:53] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:22:59] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:23:03] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:23:03] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:23:05] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:23:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:23:29] PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:23:49] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:23:51] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:23:52] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:23:59] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:24:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:24:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:24:50] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:25:32] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:25:40] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:26:02] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:26:39] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:26:52] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:27:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:27:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:27:19] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:27:21] PROBLEM - Puppet run on tools-logs-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:27:27] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:27:39] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:27:39] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:27:55] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:27:57] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:28:05] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:28:07] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:28:24] PROBLEM - Puppet run on tools-static-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:28:28] PROBLEM - Puppet run on tools-exec-1211 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:28:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:28:45] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:30:15] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:30:25] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:31:13] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:31:22] There seems to be puppet error chasemp Krenair andrewbogott ^^ [15:31:35] I'm fixing it [15:31:46] Ok thanks [15:32:05] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:32:15] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:32:27] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:32:37] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:32:37] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:32:41] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:32:41] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:32:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:33:23] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:33:31] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:33:39] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:33:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:35:28] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:35:52] PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:36:12] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:37:08] PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:37:10] PROBLEM - Puppet run on tools-exec-1206 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:38:14] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:38:14] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:38:26] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:38:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:39:16] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:39:38] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:39:46] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:39:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:40:14] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:40:25] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:40:25] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:41:09] PROBLEM - Puppet run on tools-prometheus-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:53:02] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:36] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:38] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:58] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:30] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:50] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:52] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:04] RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:14] RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:20] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:50] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:10] RECOVERY - Puppet run on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:12] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:09] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:11] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:11] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:33] RECOVERY - Puppet run on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:57] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:59] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:03] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:03] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:15] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:23] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:39] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:41] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:42] RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:43] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:47] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:51] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [15:59:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [15:59:25] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:59:31] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:24] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:42] RECOVERY - Puppet run on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:58] RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:18] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:18] RECOVERY - Puppet run on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:18] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:46] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:58] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:04] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:48] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:52] RECOVERY - Puppet run on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:00] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:49] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:25] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:33] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:39] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:59] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:13] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:39] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:51] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:13] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:23] RECOVERY - Puppet run on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:29] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:39] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:55] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:56] RECOVERY - Puppet run on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:06] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:28] RECOVERY - Puppet run on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:28] RECOVERY - Puppet run on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:30] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:38] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:48] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [16:10:16] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [16:11:12] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:02] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:08] RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:10] RECOVERY - Puppet run on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:26] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:35] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:36] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:40] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:41] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:17] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:23] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:31] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:37] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:49] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:45] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:15] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:27] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:55] RECOVERY - Puppet run on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:23] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms [16:18:13] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:26] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [16:19:14] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [16:19:36] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:26] RECOVERY - Puppet run on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [16:23:52] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:24:56] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:26:31] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:26:39] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:27:02] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:28:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:28:17] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:29:26] I will let you know when I see codezee around here [16:29:26] @notify codezee [16:30:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:31:09] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 55.76 ms [16:33:31] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [16:37:28] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [16:41:04] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [16:55:56] good morning! I'm trying to access https://piwik.wikimedia.org/ but my LDAP credentials don't seem to work. I have 2FA turned on on wikitech so maybe that's the problem? anyone know if that's a known issue? [16:57:17] bearloga: that is maintained by the analytics team, so best to ask them [16:58:02] yuvipanda: milimetric suggested I ask about this in labs/ops :) [16:58:57] ah. I think there's specifically no ops support for the piwiki service, so you might be out of luck :) [16:59:21] but no, there isn't any known issue right now with LDAP afaict. [16:59:30] bearloga, which of the login prompts? [16:59:32] bearloga: can you try using your credentials on grafana-labs-admin.wikimedia.org? [17:00:51] Krenair: the first one. yuvipanda: credentials work [17:01:21] ok! then LDAP is working fine :) [17:01:26] :) [17:03:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:18] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:33] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:35] Krenair yuvipanda: Dan's gonna look into it later. glad things are OK with LDAP & 2FA :D [17:06:37] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:54] bearloga: cool! \o/ [17:06:57] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:04] 06Labs, 10Tool-Labs: Setup Failoverable Puppetmasters for tools - https://phabricator.wikimedia.org/T145883#2644019 (10yuvipanda) [17:20:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:22:46] RECOVERY - Puppet run on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [18:08:52] PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:07:16] 06Labs, 10Labs-Infrastructure, 07LDAP: LDAP contains two extra incorrect host entries with aRecord=10.68.17.118, one with aRecord=10.68.22.5, and one with aRecord=10.68.16.120 - https://phabricator.wikimedia.org/T134025#2644583 (10AlexMonk-WMF) parsoid05 is now gone from LDAP - @hashar created a new instance... [21:44:33] 06Labs, 10Labs-Infrastructure, 07LDAP: LDAP contains two extra incorrect host entries with aRecord=10.68.17.118, one with aRecord=10.68.22.5, and one with aRecord=10.68.16.120 - https://phabricator.wikimedia.org/T134025#2644630 (10hashar) Anything from the `contintcloud` project can be deleted on spot. I am... [21:45:03] Krenair: I am pretty sure the wmflabs provisioning leaked on deletion of instances [21:45:10] and failed to delete LDAP / DNS entries somehow [21:45:36] it might be gone now. On https://phabricator.wikimedia.org/T115194 I even asked to drop all the old entries [21:53:34] good week-end all [22:02:41] 06Labs: Update custom sink handlers for Designate Mitaka - https://phabricator.wikimedia.org/T134280#2260689 (10AlexMonk-WMF) oslo_log import already looks okay: https://gerrit.wikimedia.org/r/#/c/300455/2/nova_fixed_multi/base.py