[00:16:55] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #398: 04FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/398/ [00:17:00] Project selenium-Flow » firefox,beta,Linux,BrowserTests build #398: 04FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/398/ [00:26:58] PROBLEM - Puppet staleness on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [00:45:56] (03PS1) 10Gergő Tisza: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 [00:46:18] (03PS2) 10Gergő Tisza: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) [00:49:06] (03CR) 10jerkins-bot: [V: 04-1] Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [01:00:27] (03CR) 10Gergő Tisza: "I can't make sense of the errors." [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [02:21:55] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3283293 (10Amqui) Good day, I would like to add some of my observations working with contr... [04:06:31] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #399: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/399/ [07:00:49] (03PS1) 10Legoktm: [WIP] Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) [07:01:35] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [07:20:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:14:20] (03PS2) 10Legoktm: Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) [08:14:44] 10MediaWiki-Codesniffer, 13Patch-For-Review: Update squizlabs/PHP_CodeSniffer to 3.x - https://phabricator.wikimedia.org/T142474#3283481 (10Legoktm) a:03Legoktm [08:17:29] (03CR) 10Jforrester: [C: 031] "Nice upstream changes here." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [08:30:09] (03PS3) 10Legoktm: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [08:32:48] (03CR) 10jerkins-bot: [V: 04-1] Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [08:52:06] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:53:59] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:05:55] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 06Reading-Web-Backlog, 15User-zeljkofilipin: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940#3283559 (10Jdlrobson) @greg do we need to tag this since it relates to an incident? [09:22:34] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Backlog), 10DBA, 06Operations, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3283618 (10jcrespo) I also use colors, check my bashrc if you want them: {F8158232} [09:28:12] 05Gerrit-Migration, 07Epic: Align basics of current Gerrit code-review process with Differential - https://phabricator.wikimedia.org/T167#3283632 (10TerraCodes) > Inheritance can can be set up. For example, permissions for mediawiki/extensions inherit from mediawiki, and permissions for mediawiki/extensions/Gu... [09:32:06] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:33:58] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [09:51:00] 10MediaWiki-Codesniffer, 13Patch-For-Review: Update squizlabs/PHP_CodeSniffer to 3.x - https://phabricator.wikimedia.org/T142474#3283648 (10Paladox) @legoktm thanks for working on this :) [10:08:50] Hi. Who from RelEng is around for a couple of very quick Qs? [10:15:53] Elitre: hello :) [10:16:06] ahoy! [10:16:17] Elitre: I'm just walking to lunch and will be in the tech-mgt offsite afterwards, email? [10:21:27] (03PS1) 10Hashar: Add jobs for mediawiki/extensions/Whoops [integration/config] - 10https://gerrit.wikimedia.org/r/355090 [10:22:02] (03CR) 10Hashar: "That would run the PHPUnit tests using mediawiki/vendor, I guess we should use composer instead." [integration/config] - 10https://gerrit.wikimedia.org/r/355090 (owner: 10Hashar) [10:22:49] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3283728 (10Elitre) [10:23:05] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:23:34] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3199999 (10Elitre) (If I understood Greg's directions correctly, T37247 belongs here.) [10:28:26] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3282853 (10hashar) The `mwdeploy` user requires (or required at some point) to have the same UID on all the deployment target. To achieve that, we went to add the... [10:32:51] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 06Release-Engineering-Team (Kanban), 13Patch-For-Review: For MediaWiki extensions, merge composer test into mwext-textextension / mediawiki-extensions jobs - https://phabricator.wikimedia.org/T161895#3283761 (10hashar) [10:33:40] 10Continuous-Integration-Infrastructure, 10Page-Previews, 06Reading-Web-Backlog: Execute mwgate-composer workers in CI while building Popups extension - https://phabricator.wikimedia.org/T165521#3268886 (10hashar) Indeed the `composer test` step has been merged with the jobs running PHPUnit tests. Happened... [10:34:24] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) [10:44:25] Project beta-scap-eqiad build #156368: 04FAILURE in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156368/ [10:44:43] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:45:55] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:49:24] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [10:50:19] 06Release-Engineering-Team, 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3283782 (10Qgil) What if #release-engineering-team would have veto power for any API breaks that haven't be... [10:54:31] Project beta-scap-eqiad build #156369: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156369/ [10:59:33] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:03:06] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:04:18] Project beta-scap-eqiad build #156370: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156370/ [11:14:19] Project beta-scap-eqiad build #156371: 04STILL FAILING in 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156371/ [11:24:17] Project beta-scap-eqiad build #156372: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156372/ [11:34:18] Project beta-scap-eqiad build #156373: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156373/ [11:35:31] ugh [11:40:35] !log gjg@deployment-tin:/srv/mediawiki/.git/gc.log has warning: There are too many unreachable loose objects; run 'git prune' to remove them. [11:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:44:25] Project beta-scap-eqiad build #156374: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156374/ [11:54:20] Project beta-scap-eqiad build #156375: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156375/ [11:58:20] (03CR) 10Gergő Tisza: "Thanks! I tried the same thing in Iac1de79bba667b0a79b8617b3ad6867a4befa367 (at least I thought it was the same thing :)" [integration/config] - 10https://gerrit.wikimedia.org/r/355090 (owner: 10Hashar) [12:04:20] Project beta-scap-eqiad build #156376: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156376/ [12:08:11] greg-g: Did you run `git prune`? [12:10:13] I *think* that repo is just testing the scap to git stuff. Not sure that it will cause problems [12:10:27] sudo -u mwdeploy git prune ? [12:10:38] Plausibly? [12:11:15] gjg@deployment-tin:/srv/mediawiki$ sudo -u mwdeploy git prune [12:11:16] Checking connectivity: 393006, done. [12:11:55] !log ran git prune and rm'd the gc.log file [12:11:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:13:53] * greg-g waits for the cron while talking quarterly planning [12:16:15] Yippee, build fixed! [12:16:15] Project beta-scap-eqiad build #156377: 09FIXED in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156377/ [12:17:18] Yay. [12:17:22] Thanks greg-g! [12:19:31] weeee [12:19:47] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3283971 (10Jdforrester-WMF) [12:20:27] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3199999 (10Jdforrester-WMF) [12:21:55] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3283984 (10Jdforrester-WMF) [12:22:19] And that's all three remaining VE blockers to wmf.2 provisionally fixed. So yay. [12:22:29] w00t! [12:22:56] (Wanted to test in Beta Cluster as well as locally, hence being irritated by beta-scap-eqiad. ;-)) [12:25:14] figured :) [13:24:39] PROBLEM - Puppet errors on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:38:53] PROBLEM - Puppet errors on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:40:11] PROBLEM - Puppet errors on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:44:51] PROBLEM - Puppet errors on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:58:01] 06Release-Engineering-Team, 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3284192 (10Tgr) >>! In T114384#3283782, @Qgil wrote: > What if #release-engineering-team would have veto po... [14:04:20] (03PS4) 10Hashar: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:04:53] (03Abandoned) 10Hashar: Add jobs for mediawiki/extensions/Whoops [integration/config] - 10https://gerrit.wikimedia.org/r/355090 (owner: 10Hashar) [14:06:11] (03CR) 10jerkins-bot: [V: 04-1] Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:07:16] (03PS5) 10Hashar: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:11:11] (03CR) 10Hashar: [C: 032] Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:11:19] (03CR) 10Gergő Tisza: "Oh, duh, spacing. Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:12:47] (03Merged) 10jenkins-bot: Set up CI for Whoops extension [integration/config] - 10https://gerrit.wikimedia.org/r/355063 (https://phabricator.wikimedia.org/T111731) (owner: 10Gergő Tisza) [14:18:10] 10Continuous-Integration-Infrastructure, 10Deployment-Systems, 06Release-Engineering-Team (Backlog): Unify deployment of integration/config.git changes using the official Wikimedia deployment system - https://phabricator.wikimedia.org/T111559#3284237 (10hashar) [14:18:12] 10Continuous-Integration-Infrastructure, 10Scap (Scap3-Adoption-Phase1): Migrate Zuul deployment of integration/config to scap deploy - https://phabricator.wikimedia.org/T129357#3284235 (10hashar) 05Open>03declined Will stick to deb packages for now. There is only a couple host that relies on it and we on... [14:18:36] 10Continuous-Integration-Infrastructure, 10Deployment-Systems, 06Release-Engineering-Team (Backlog): Unify deployment of integration/config.git changes using the official Wikimedia deployment system - https://phabricator.wikimedia.org/T111559#1610383 (10hashar) [14:18:38] 10Continuous-Integration-Infrastructure, 10Scap (Scap3-Adoption-Phase1): Migrate Zuul deployment of integration/config to scap deploy - https://phabricator.wikimedia.org/T129357#3284238 (10hashar) 05declined>03Open Reopening, wrong task / I am confused. [14:29:16] 06Release-Engineering-Team (Watching / External), 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3284248 (10greg) >>! In T114384#3284192, @Tgr wrote: > So what you are saying really... [14:33:42] PROBLEM - Free space - all mounts on deployment-db03 is CRITICAL: CRITICAL: deployment-prep.deployment-db03.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db03.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found)deployment-prep.deployment-db03.diskspace._srv.byte_percentfree (<50.00%) [14:34:28] 10Beta-Cluster-Infrastructure: beta cluster databases have almost full disks - https://phabricator.wikimedia.org/T166060#3284262 (10hashar) [14:35:18] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta cluster databases have almost full disks - https://phabricator.wikimedia.org/T166060#3284274 (10hashar) [14:38:23] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 06Release-Engineering-Team: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3284277 (10hashar) [15:47:53] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3284426 (10Ladsgroup) Most of it is being done but there are two things needing to be done: - Update packagist, I don't... [15:53:28] (03PS1) 10Ladsgroup: Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) [15:55:17] (03CR) 10jerkins-bot: [V: 04-1] Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) (owner: 10Ladsgroup) [15:59:03] (03PS2) 10Ladsgroup: Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) [16:04:11] 06Release-Engineering-Team (Kanban), 05MW-1.29-release (WMF-deploy-2016-11-01_(1.29.0-wmf.1)), 13Patch-For-Review: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#3284479 (10demon) If we update docs to tell people to use `~/.config/git-review/git-review.conf`, we c... [16:05:36] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3284494 (10Tobi_WMDE_SW) I guess we would also need to somehow also port the releases over to Gerrit, no? [16:12:55] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3284545 (10Ladsgroup) We can also go with dev-master like Wikibase and other gerrit extensions. [16:40:55] (03CR) 10Paladox: [C: 031] "+1 too" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [17:05:39] PROBLEM - Puppet errors on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:05:44] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:19] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:16:12] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:28:53] 10Continuous-Integration-Config, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Kanban-Board, 07Unplanned-Sprint-Work: Migrate experimental CI job mwext-MobileFrontend-npm-run-lint-modules to stable - https://phabricator.wikimedia.org/T163882#3213309 (10pmiazga) I'll sign off this task [17:29:03] 10Continuous-Integration-Config, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Kanban-Board, 07Unplanned-Sprint-Work: Migrate experimental CI job mwext-MobileFrontend-npm-run-lint-modules to stable - https://phabricator.wikimedia.org/T163882#3284717 (10pmiazga) a:03pmiazga [17:41:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [17:51:10] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:10:49] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#1413188 (10greg) Looks like the gerrit repo admin part is done, right? Can I remove releng-related projects now? [20:15:32] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3284987 (10Ladsgroup) @greg: Yes, thank you for doing it. [20:25:40] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3285013 (10Legoktm) >>! In T104309#3284494, @Tobi_WMDE_SW wrote: > I guess we would also need to somehow also port the r... [21:02:59] 24 [21:39:53] 25 [22:16:20] PROBLEM - Free space - all mounts on deployment-db04 is CRITICAL: CRITICAL: deployment-prep.deployment-db04.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db04.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found)deployment-prep.deployment-db04.diskspace._srv.byte_percentfree (<30.00%) [22:28:11] We should kill some of the simplewiki DB [22:28:12] it's too big [22:28:28] Er, maybe some other stuff [22:32:16] updatelog is huge [22:32:19] Cuz yay, update.php [22:32:33] Like 600mb-1gb each wiki [22:32:43] Just saved ~40gb [22:32:46] Should recover [22:41:19] RECOVERY - Free space - all mounts on deployment-db04 is OK: OK: deployment-prep.deployment-db04.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db04.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found) [22:43:42] RECOVERY - Free space - all mounts on deployment-db03 is OK: OK: deployment-prep.deployment-db03.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db03.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found) [23:20:37] Project beta-update-databases-eqiad build #17268: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17268/ [23:21:28] Undefined index: mediaSwiftAuth [23:21:52] Caused by https://gerrit.wikimedia.org/r/#/c/353173/ [23:22:04] RainbowSprinkles ^^ [23:34:08] Reverted. [23:45:20] What's this new bot? [23:45:20] :p [23:45:25] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]