[00:04:18] harej: clearly you need to start a change.org petition >.> [00:23:44] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:03:41] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:06:05] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3403069 (10Dzahn) release files are now auto-rsynced: ``` [releases1001:~] $ sudo crontab -l | grep releases *... [01:42:23] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:43:31] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3403113 (10Dzahn) permission issue fixed ^ , looks like this, just like on bromine, and also stays like that afte... [01:56:24] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:22:24] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:31:23] RECOVERY - Puppet errors on castor is OK: OK: Less than 1.00% above the threshold [0.0] [02:43:38] Project beta-scap-eqiad build #162446: 04FAILURE in 0.36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162446/ [02:55:57] Yippee, build fixed! [02:55:58] Project beta-scap-eqiad build #162447: 09FIXED in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162447/ [03:18:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:53:35] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:58:33] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #442: 04FAILURE in 2 min 33 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/442/ [04:37:51] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3403249 (10demon) >>! In T157414#3402741, @Paladox wrote: > What about having a script on the client that moves the file into place after scapping. Ie like we do... [04:45:36] Project beta-scap-eqiad build #162458: 04FAILURE in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162458/ [04:55:57] Yippee, build fixed! [04:55:58] Project beta-scap-eqiad build #162459: 09FIXED in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162459/ [06:33:10] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [06:40:38] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:41:51] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:42:01] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:42:19] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:42:19] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:42:24] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:42:30] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:42:38] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:42:40] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:42:40] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:42:50] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:43:09] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:43:24] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:43:41] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:44:19] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:44:27] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:44:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:44:33] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:44:48] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:47:45] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:48:07] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:48:40] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:48:47] PROBLEM - Puppet errors on deployment-salt02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:49:38] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:50:04] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:52:04] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:52:28] PROBLEM - Puppet errors on deployment-poolcounter04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:52:45] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:52:47] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:53:45] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:53:47] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:54:12] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:54:40] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:54:42] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:55:54] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:56:18] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:57:25] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:57:59] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:58:18] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:58:40] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:58:42] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:59:16] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:59:45] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:00:44] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:01:00] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:01:48] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:01:48] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:02:04] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:02:04] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:02:35] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:02:35] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:03:01] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:03:01] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:04:12] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:04:12] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:04:22] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:04:24] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:04:28] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:04:28] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:04:52] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:04:52] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:06:00] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:06:00] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:07:03] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:07:03] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:11:35] 10Continuous-Integration-Infrastructure: Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602#3403407 (10Volans) [08:21:27] (03CR) 10Hashar: [C: 04-2] "There is no guarantee it would always be executed. Instead it should trigger another job that does the cleanup on the publishing host." [integration/config] - 10https://gerrit.wikimedia.org/r/363047 (owner: 10Hashar) [08:38:21] (03PS2) 10Hashar: Move wikimedia-fundraising-civicrm to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/319331 [09:10:59] (03CR) 10Hashar: [C: 04-1] "https://gerrit.wikimedia.org/r/#/c/363141/ points sendmail as /bin/true" [integration/config] - 10https://gerrit.wikimedia.org/r/319331 (owner: 10Hashar) [09:26:05] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-Nonlinear, 10Project-Admins, 10Patch-For-Review: Archive the NonLinear extension - https://phabricator.wikimedia.org/T169519#3400422 (10Nemo_bis) I'd prefer that bugs be closed only *after* the repositories are emptied, otherwise they'll k... [09:28:28] !log gerrit: marking read-only mediawiki/extensions/Nonlinear - T169519 [09:28:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:28:34] T169519: Archive the NonLinear extension - https://phabricator.wikimedia.org/T169519 [09:28:50] (03CR) 10Hashar: [C: 032] Archive Nonlinear [integration/config] - 10https://gerrit.wikimedia.org/r/362973 (https://phabricator.wikimedia.org/T169519) (owner: 10MarcoAurelio) [09:30:15] (03Merged) 10jenkins-bot: Archive Nonlinear [integration/config] - 10https://gerrit.wikimedia.org/r/362973 (https://phabricator.wikimedia.org/T169519) (owner: 10MarcoAurelio) [10:52:06] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3403914 (10phuedx) I would note that Frankiebot overwrites the topic of the original change. It'd... [11:05:21] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:40:20] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:31:29] hashar: looks like puppet is failing on most hosts in deployment-prep, not sure if known already [13:32:36] godog: indeed :( http://shinken.wmflabs.org/problems?search=deployment [13:32:38] will check [13:33:13] puppet master is dead [13:33:22] !log beta cluster puppet is broken: Error: Could not send report: Connection refused - connect(2) for "deployment-puppetmaster02.deployment-prep.eqiad.wmflabs" port 8140 [13:33:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:34:01] Jul 04 06:36:14 deployment-puppetmaster02 apache2[5533]: Invalid command 'SSLOpenSSLConfCmd', perhaps misspelled or defined by a module not included in the server configuration [13:34:02] bah [13:34:24] caused by https://phabricator.wikimedia.org/T159254 [13:44:43] hashar does the wikimedia repo work on those hosts? [13:45:04] I found when the repo is unreachable apache2 update is offered from the debian repo which will break things [13:48:29] 10Gerrit, 10Operations: rename user gerrit2 to gerrit - https://phabricator.wikimedia.org/T169634#3404360 (10Paladox) [13:58:23] godog: TLDR: unattended upgrade as its own custom apt policy :-((( [14:07:33] hashar: doh, thanks I've subscribed to that bug [14:08:08] godog: I have spammed it with bunch of things [14:08:17] but in short unattend upgrade configuration has to be tweaked [14:08:22] the CI puppet master has not been affected [14:08:32] I guess moritz will parse my writings eventually :-} [14:09:14] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:18] or we could use openssl from jessie-backports :-} But I am not sure whether it is still maintained [14:09:25] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:29] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:43] RECOVERY - Puppet errors on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:10:13] or upgrade to stretch! [14:10:18] ;-] [14:10:43] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [14:10:56] !log manually upgraded apache2 on deployment-puppetmaster02 see T159254 [14:11:00] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:11:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:11:01] T159254: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254 [14:11:48] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:04] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:26] RECOVERY - Puppet errors on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:32] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:13:01] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:51] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [14:16:01] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [14:16:07] godog: and it is recovering just fine (had to manually upgrade, but unattended upgrade will hit again tomorrow) [14:16:51] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:03] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:17] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:19] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:30] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:15] hashar: I guess you can hold apache until there's a solution, or disable unattended-upgrade on the master [14:18:42] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:18] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:34] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:46] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:00] godog: I have reopened a few old patches that are still applied on the CI puppetmaster. That apparently did the trick for that host [14:20:38] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:00] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:02] godog: https://gerrit.wikimedia.org/r/#/c/315084 which might or might not fix it [14:22:20] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:36] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:38] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:40] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:47] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:04] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:28] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:34] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:30] RECOVERY - Puppet errors on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:42] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:44] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:41] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:45] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:13] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:37] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:43] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:08] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:56] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:18] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:04] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:44] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:47] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:19] RECOVERY - Puppet errors on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:43] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:02] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:16] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:45] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:39:09] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage gerrit2001 and cobalt as stretch - https://phabricator.wikimedia.org/T168562#3404595 (10Paladox) [14:40:32] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3404597 (10Paladox) gerrit will change it's host key when we upgrade to gerrit 2.... [14:44:25] 10Gerrit: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148#3404624 (10Paladox) [14:44:27] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3404623 (10Paladox) [15:09:13] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:09:43] 10Continuous-Integration-Infrastructure (phase-out-trusty): Install PHP5.5 on jessie CI instances - https://phabricator.wikimedia.org/T144959#3404752 (10Paladox) http://packages.dotdeb.org has php5.5 packages. We should try to run multi php on jessie. Will allow more jessie tests to run. [15:17:21] (03PS2) 10Hashar: (DO NOT SUBMIT) experimental R based job [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) [15:19:50] (03CR) 10Hashar: "Changed the command line to use --vanilla which seems to pass the proper set of --no-something" [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [15:24:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:53:27] (03PS2) 10Addshore: Add Newsletter to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/362388 (https://phabricator.wikimedia.org/T110170) [15:55:30] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #476: 04FAILURE in 33 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/476/ [16:04:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:11] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:14:27] 10Beta-Cluster-Infrastructure, 10WikimediaMessages: Special:ListGroupRights shows "extendedmover" literally on enwiki beta labs - https://phabricator.wikimedia.org/T169663#3405051 (10GeoffreyT2000) [16:20:34] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:21:44] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3405073 (10Dzahn) [16:24:39] 10Gerrit, 10Release-Engineering-Team, 10Operations: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3405078 (10Dzahn) One at a time please, first gerrit2001 only i suggest. [17:08:11] (03CR) 10Reedy: [C: 032] Add Newsletter to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/362388 (https://phabricator.wikimedia.org/T110170) (owner: 10Addshore) [17:09:00] (03Merged) 10jenkins-bot: Add Newsletter to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/362388 (https://phabricator.wikimedia.org/T110170) (owner: 10Addshore) [17:10:00] (03PS1) 10Addshore: Add extension-phan-generic to FileImporter [integration/config] - 10https://gerrit.wikimedia.org/r/363220 [17:29:46] (03PS1) 10Addshore: Use phan version 0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/363222 [17:40:35] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services, 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3405238 (10Addshore) In Fact, coming back to this task and reading point 1 in the list I really think this effort ha... [18:43:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:54:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Upgrade tox on CI instances - https://phabricator.wikimedia.org/T169602#3405375 (10hashar) It is probably best to stick to the Debian packages. While at it we should do the same for `pip` and `setuptools`. They are all installed fro... [18:58:11] (03CR) 10Hashar: "Isn't the mwreleases server going to be a Jenkins? If so you could just git clone the repo as part of the job and save yourself the troub" [tools/release] - 10https://gerrit.wikimedia.org/r/356430 (owner: 10Chad) [19:04:45] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-Nonlinear, 10Project-Admins, 10Patch-For-Review: Archive the NonLinear extension - https://phabricator.wikimedia.org/T169519#3405383 (10MarcoAurelio) [19:05:19] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-Nonlinear, 10Project-Admins, 10Patch-For-Review: Archive the NonLinear extension - https://phabricator.wikimedia.org/T169519#3400422 (10MarcoAurelio) [19:14:43] Project beta-scap-eqiad build #162546: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162546/ [19:23:41] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:30] Yippee, build fixed! [19:24:31] Project beta-scap-eqiad build #162547: 09FIXED in 9 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162547/ [19:38:37] (03PS1) 10Ejegg: Add civicrm-buildkit/bin to path for CRM builds [integration/config] - 10https://gerrit.wikimedia.org/r/363229 (https://phabricator.wikimedia.org/T169593) [19:44:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:19:44] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:01:45] (03CR) 10Eileen: [C: 031] "This makes sense to me, apparently I can't approve it." [integration/config] - 10https://gerrit.wikimedia.org/r/363229 (https://phabricator.wikimedia.org/T169593) (owner: 10Ejegg) [21:55:55] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:24:48] (03PS2) 10Ejegg: Add civicrm-buildkit/bin to path for CRM builds [integration/config] - 10https://gerrit.wikimedia.org/r/363229 (https://phabricator.wikimedia.org/T169593) [23:25:45] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]