[00:06:45] <wmf-insecte>	 Project beta-scap-eqiad build #286291: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286291/
[00:07:14] <wikibugs>	 10Beta-Cluster-Infrastructure: Host key verification failed x 2 - https://phabricator.wikimedia.org/T244074 (10Reedy)
[00:16:35] <wmf-insecte>	 Project beta-scap-eqiad build #286292: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286292/
[00:26:36] <wmf-insecte>	 Project beta-scap-eqiad build #286293: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286293/
[00:36:34] <wmf-insecte>	 Project beta-scap-eqiad build #286294: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286294/
[00:46:32] <wmf-insecte>	 Project beta-scap-eqiad build #286295: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286295/
[00:56:28] <wmf-insecte>	 Project beta-scap-eqiad build #286296: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286296/
[01:06:45] <wmf-insecte>	 Project beta-scap-eqiad build #286297: 04STILL FAILING in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286297/
[01:16:37] <wmf-insecte>	 Project beta-scap-eqiad build #286298: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286298/
[01:26:35] <wmf-insecte>	 Project beta-scap-eqiad build #286299: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286299/
[01:36:34] <wmf-insecte>	 Project beta-scap-eqiad build #286300: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286300/
[01:46:37] <wmf-insecte>	 Project beta-scap-eqiad build #286301: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286301/
[01:56:37] <wmf-insecte>	 Project beta-scap-eqiad build #286302: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286302/
[02:06:42] <wmf-insecte>	 Project beta-scap-eqiad build #286303: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286303/
[02:16:38] <wmf-insecte>	 Project beta-scap-eqiad build #286304: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286304/
[02:26:30] <wmf-insecte>	 Project beta-scap-eqiad build #286305: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286305/
[02:36:36] <wmf-insecte>	 Project beta-scap-eqiad build #286306: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286306/
[02:46:27] <wmf-insecte>	 Project beta-scap-eqiad build #286307: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286307/
[02:56:33] <wmf-insecte>	 Project beta-scap-eqiad build #286308: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286308/
[03:06:39] <wmf-insecte>	 Project beta-scap-eqiad build #286309: 04STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286309/
[03:16:38] <wmf-insecte>	 Project beta-scap-eqiad build #286310: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286310/
[03:26:36] <wmf-insecte>	 Project beta-scap-eqiad build #286311: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286311/
[03:36:44] <wmf-insecte>	 Project beta-scap-eqiad build #286312: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286312/
[03:46:46] <wmf-insecte>	 Project beta-scap-eqiad build #286313: 04STILL FAILING in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286313/
[03:53:59] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[03:56:42] <wmf-insecte>	 Project beta-scap-eqiad build #286314: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286314/
[04:06:52] <wmf-insecte>	 Project beta-scap-eqiad build #286315: 04STILL FAILING in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286315/
[04:16:47] <wmf-insecte>	 Project beta-scap-eqiad build #286316: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286316/
[04:26:46] <wmf-insecte>	 Project beta-scap-eqiad build #286317: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286317/
[04:36:42] <wmf-insecte>	 Project beta-scap-eqiad build #286318: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286318/
[04:46:44] <wmf-insecte>	 Project beta-scap-eqiad build #286319: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286319/
[04:56:32] <wmf-insecte>	 Project beta-scap-eqiad build #286320: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286320/
[05:03:50] <wikibugs>	 10Continuous-Integration-Config, 10VideoCutTool: Set up continuous integration for VideoCutTool - https://phabricator.wikimedia.org/T244079 (10Gopavasanth)
[05:06:49] <wmf-insecte>	 Project beta-scap-eqiad build #286321: 04STILL FAILING in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286321/
[05:16:41] <wmf-insecte>	 Project beta-scap-eqiad build #286322: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286322/
[05:26:28] <wmf-insecte>	 Project beta-scap-eqiad build #286323: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286323/
[05:30:38] <wikibugs>	 (03PS1) 10Gopavasanth: Add labs/tools/VideoCutTool [integration/config] - 10https://gerrit.wikimedia.org/r/569352 (https://phabricator.wikimedia.org/T244079)
[05:31:24] <wikibugs>	 (03PS2) 10Gopavasanth: Add labs/tools/VideoCutTool [integration/config] - 10https://gerrit.wikimedia.org/r/569352 (https://phabricator.wikimedia.org/T244079)
[05:36:39] <wmf-insecte>	 Project beta-scap-eqiad build #286324: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286324/
[05:46:44] <wmf-insecte>	 Project beta-scap-eqiad build #286325: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286325/
[05:56:44] <wmf-insecte>	 Project beta-scap-eqiad build #286326: 04STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286326/
[06:06:56] <wmf-insecte>	 Project beta-scap-eqiad build #286327: 04STILL FAILING in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286327/
[06:16:46] <wmf-insecte>	 Project beta-scap-eqiad build #286328: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286328/
[06:26:50] <wmf-insecte>	 Project beta-scap-eqiad build #286329: 04STILL FAILING in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286329/
[06:36:43] <wmf-insecte>	 Project beta-scap-eqiad build #286330: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286330/
[06:43:47] <wmf-insecte>	 Project beta-scap-eqiad build #286331: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286331/
[06:47:33] <wmf-insecte>	 Project beta-scap-eqiad build #286332: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286332/
[06:56:55] <wmf-insecte>	 Yippee, build fixed!
[06:56:55] <wmf-insecte>	 Project beta-scap-eqiad build #286333: 09FIXED in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286333/
[06:57:15] <thcipriani|afk>	 !log deployment-deploy01$ while read host; do ssh-keyscan -H "$host"; done < <(grep -vP '^#' /etc/dsh/group/mediawiki-installation | grep -vP '^$' ) | sudo -u jenkins-deploy tee /mnt/home/jenkins-deploy/.ssh/known_hosts
[06:57:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:14:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<100.00%)
[10:34:25] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-logstash03 is OK: OK: All targets OK
[10:43:05] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-logstash2 is OK: OK: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)
[12:28:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) @jbond: Okay so I think our puppetdb stuff is migrated now. I just removed the old server from the list, removed command broadcasting, and puppet still seems to be working. Nex...
[12:38:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Krenair) worked around by deploying the new cert from acme-chief manually on -cache-text05 for now. upload may still be br...
[12:39:40] <wikibugs>	 10Beta-Cluster-Infrastructure: Host key verification failed x 2 - https://phabricator.wikimedia.org/T244074 (10Krenair) This may be due to stuff going on in T243226. -mediawiki-07 should be better, I think snapshot01 needs more work to handle its special puppetmaster
[12:46:57] <wikibugs>	 10Beta-Cluster-Infrastructure: Host key verification failed x 2 - https://phabricator.wikimedia.org/T244074 (10Krenair) a:03Krenair https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/4cecb2fbef2ea8a54e7e4134a655d7b37d900228%5E%21/ -> ` krenair@deployment-dumps-puppetmaster02:~$ sudo  puppet...
[12:59:06] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<55.56%)
[13:37:34] <wikibugs>	 (03PS1) 10MarcoAurelio: Modify access rules [extensions/AControlImageLink] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569357
[13:37:40] <wikibugs>	 (03Abandoned) 10MarcoAurelio: Modify access rules [extensions/AControlImageLink] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569357 (owner: 10MarcoAurelio)
[13:41:57] <wikibugs>	 (03PS1) 10MarcoAurelio: Modify access rules [extensions/AControlImageLink] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569359
[13:42:33] <wikibugs>	 (03PS2) 10MarcoAurelio: Modify access rules [extensions/AControlImageLink] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569359
[13:51:39] <wikibugs>	 (03PS3) 10MarcoAurelio: Set `mediawiki/extensions` as parent project [extensions/AControlImageLink] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569359
[18:38:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) Just done the puppetmaster upgrade and found puppet no longer runs on the puppetmaster: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error:...
[19:10:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<22.22%)
[19:11:59] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[19:24:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) Is it possibly our lack of updated puppetdb-termini package? `root@deployment-puppetmaster03:~# apt-cache policy puppetdb-termini puppetdb-termini:   Installed: 4.4.0-1~wmf2...
[19:46:25] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10DannyS712)
[20:50:43] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Xaosflux) {F31545291}
[20:51:28] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Xaosflux) Note on the screen shot from Cyphoidbomb's failure above, this is referencing a group that doesn't ev...
[21:06:51] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[21:22:14] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Masumrezarock100) FlaggedRevs is buggy and has no maintainer currently. A bug like this isn't so surprising. Bu...