[05:08:02] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [05:12:59] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [07:00:27] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [07:10:26] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [07:20:59] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) 05Open→03Resolved Dropped `reviewdb` after double checking nothing wrote to it aga... [07:59:02] 10Phabricator, 10Mail: Duplicate weekly Phabricator emails (cronjob ran twice?) - https://phabricator.wikimedia.org/T258371 (10Aklapper) p:05Triage→03Low [08:28:28] PROBLEM - Host deployment-changeprop is DOWN: CRITICAL - Host Unreachable (172.16.5.21) [08:28:28] PROBLEM - Host deployment-cpjobqueue is DOWN: CRITICAL - Host Unreachable (172.16.4.124) [08:28:28] PROBLEM - Host deployment-chromium02 is DOWN: CRITICAL - Host Unreachable (172.16.4.14) [08:37:35] (03CR) 10Jforrester: [C: 03+1] Add PSR2.ControlStructures.SwitchDeclaration [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/613734 (https://phabricator.wikimedia.org/T182546) (owner: 10Esanders) [08:43:23] (03CR) 10Jforrester: "s/allows/forces/, right? It'll no longer parse .inc files with this change? We still have .inc files in MW core…" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/614285 (https://phabricator.wikimedia.org/T200956) (owner: 10Umherirrender) [09:50:44] maintenance-disconnect-full-disks build 195777 integration-agent-docker-1013 (/srv: 100%): OFFLINE due to disk space [09:52:05] PROBLEM - Free space - all mounts on integration-agent-docker-1013 is CRITICAL: CRITICAL: integration.integration-agent-docker-1013.diskspace._srv.byte_percentfree (<44.44%) [10:02:06] RECOVERY - Free space - all mounts on integration-agent-docker-1013 is OK: OK: All targets OK [10:05:28] maintenance-disconnect-full-disks build 195780 integration-agent-docker-1013: OFFLINE due to disk space [10:30:28] maintenance-disconnect-full-disks build 195785 integration-agent-docker-1013: OFFLINE due to disk space [10:55:41] maintenance-disconnect-full-disks build 195790 integration-agent-docker-1013: OFFLINE due to disk space [11:00:23] 10MediaWiki-Codesniffer, 10MW-1.36-notes (1.36.0-wmf.1; 2020-07-21), 10Patch-For-Review: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546 (10thiemowmde) Personally, I disagree with making this a strict rule. There is nothing wrong with either of the two `switch`... [11:20:30] maintenance-disconnect-full-disks build 195795 integration-agent-docker-1013: OFFLINE due to disk space [11:45:26] maintenance-disconnect-full-disks build 195800 integration-agent-docker-1013: OFFLINE due to disk space [12:00:29] maintenance-disconnect-full-disks build 195803 integration-agent-docker-1011 (/srv: 99%): OFFLINE due to disk space [12:01:29] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [12:10:31] maintenance-disconnect-full-disks build 195805 integration-agent-docker-1011: OFFLINE due to disk space [12:10:32] maintenance-disconnect-full-disks build 195805 integration-agent-docker-1013: OFFLINE due to disk space [12:11:28] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [12:35:35] maintenance-disconnect-full-disks build 195810 integration-agent-docker-1011: OFFLINE due to disk space [12:35:36] maintenance-disconnect-full-disks build 195810 integration-agent-docker-1013: OFFLINE due to disk space [12:52:31] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Write script to merge key+signature files into single files in pgp-public-keys repository - https://phabricator.wikimedia.org/T258271 (10LarsWirzenius) p:05Triage→03Medium [12:58:15] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10observability: Extract mtail puppet.git tests to a standalone CI container - https://phabricator.wikimedia.org/T255534 (10fgiunchedi) Indeed, `mtail` is also tiny compared to th... [13:00:26] maintenance-disconnect-full-disks build 195815 integration-agent-docker-1011: OFFLINE due to disk space [13:00:27] maintenance-disconnect-full-disks build 195815 integration-agent-docker-1013: OFFLINE due to disk space [13:00:38] 10MediaWiki-Codesniffer: New Sniff for using pointless variables before return - https://phabricator.wikimedia.org/T179768 (10thiemowmde) What @Krinkle and @Tgr said. Personally, I do not use this style. I try to find other ways to make my code readable and self-explanatory. But I know certain developers (e.g. @... [13:14:23] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<11.11%) [13:24:23] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [13:25:27] maintenance-disconnect-full-disks build 195820 integration-agent-docker-1011: OFFLINE due to disk space [13:25:28] maintenance-disconnect-full-disks build 195820 integration-agent-docker-1013: OFFLINE due to disk space [13:25:41] 10MediaWiki-Codesniffer, 10MW-1.36-notes (1.36.0-wmf.1; 2020-07-21), 10Patch-For-Review: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546 (10Huji) I could say the same for how we indent the insides of a `for` loop or an `if` clause. The "there is nothing wrong" a... [13:50:27] maintenance-disconnect-full-disks build 195825 integration-agent-docker-1011: OFFLINE due to disk space [13:50:28] maintenance-disconnect-full-disks build 195825 integration-agent-docker-1013: OFFLINE due to disk space [14:00:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520 (10fgiunchedi) a:05fgiunchedi→03None Unassigning from me sin... [14:09:16] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:18:57] that's a pretty sweet 16-patch chain winding its way through ci [14:23:01] !log locking mediawiki/core for zuul merger to prevent a 16-patch chain from keeping CI occupied forever [14:23:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:23:48] !log bring integration-agent-docker-101{1,3} back online, disk space looks fine [14:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:25:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:27:05] !log unlock mediawiki/core for zuul merger [14:27:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:41:39] (03CR) 10Zfilipin: "Hi Alex and thanks for your contribution. This repository is not actively maintained. I don't have the time to do a proper review of this " [ruby/api] - 10https://gerrit.wikimedia.org/r/613666 (owner: 10Alex Dean) [14:47:28] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<22.22%) [14:57:27] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [15:21:59] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [15:46:02] 10MediaWiki-Codesniffer, 10MW-1.36-notes (1.36.0-wmf.1; 2020-07-21), 10Patch-For-Review: Require indentation of CASE statements in PHP code - https://phabricator.wikimedia.org/T182546 (10thiemowmde) > I could say the same for how we indent the insides of a `for` loop or an `if` clause. No, you can't. Removin... [15:59:06] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) Looking at contint1001 today (Mon 20 Jul 2020 03:47:28 PM UTC) I see: ` dancy@contint1001:~... [15:59:45] o/ Is there a gerrit group of people I should add for code review on the mw cli repo? or should I just keep adding kostajh ? :D [16:00:20] addshore: longma, brennen and twentyafterfour have reviewed stuff on that repo [16:00:24] and sorry for being slow! [16:00:28] kostajh: np :) [16:01:10] sorry for not managing to look back at any of it until this weekend :) My dev laptop has been on a rollercoaster of changes recently, so now I'm trying Go code writing in vscode :D [16:11:01] 10Release-Engineering-Team, 10Analytics, 10Analytics-EventLogging, 10dev-images, 10Patch-For-Review: EventLogging dev image should have verbose output enabled - https://phabricator.wikimedia.org/T257378 (10Milimetric) 05Open→03Declined moving to modern event platform, we're not going to maintain this... [16:13:05] (03CR) 10Umherirrender: "> Patch Set 1:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/614285 (https://phabricator.wikimedia.org/T200956) (owner: 10Umherirrender) [16:15:47] maintenance-disconnect-full-disks build 195854 integration-agent-docker-1007 (/srv: 99%): OFFLINE due to disk space [16:16:26] 10Release-Engineering-Team, 10Analytics-Radar, 10ChangeProp, 10Event-Platform, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583 (10Milimetric) agreed, ping us on reviews etc. [16:18:13] PROBLEM - Free space - all mounts on integration-agent-docker-1007 is CRITICAL: CRITICAL: integration.integration-agent-docker-1007.diskspace._srv.byte_percentfree (<33.33%) [16:18:32] (03CR) 10Alex Dean: "> Patch Set 2:" [ruby/api] - 10https://gerrit.wikimedia.org/r/613666 (owner: 10Alex Dean) [16:20:31] maintenance-disconnect-full-disks build 195855 integration-agent-docker-1007: OFFLINE due to disk space [16:28:11] RECOVERY - Free space - all mounts on integration-agent-docker-1007 is OK: OK: All targets OK [16:30:40] 10Release-Engineering-Team, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10kzimmerman) [16:31:37] (03CR) 10Zfilipin: "> Patch Set 2:" [ruby/api] - 10https://gerrit.wikimedia.org/r/613666 (owner: 10Alex Dean) [16:34:56] 10Release-Engineering-Team, 10Analytics, 10Analytics-EventLogging, 10dev-images, 10Patch-For-Review: EventLogging dev image should have verbose output enabled - https://phabricator.wikimedia.org/T257378 (10dbarratt) >>! In T257378#6319920, @Milimetric wrote: > moving to modern event platform, we're not g... [16:45:36] maintenance-disconnect-full-disks build 195860 integration-agent-docker-1007: OFFLINE due to disk space [16:46:32] 10Continuous-Integration-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10Cloud-Services, 10Shinken, 10WorkType-Maintenance: Labs Shinken complains about no more existing host integration-t102459 is DOWN - https://phabricator.wikimedia.org/T121767 (10Andrew) We are trying to deprecate shinken entirely, so... [16:48:30] !log integration-agent-docker-1007 back online [16:48:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:38:56] 10Release-Engineering-Team, 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10Nuria) pin here cause @calbon also needs kerberos credentials [17:41:41] PROBLEM - Free space - all mounts on integration-castor03 is CRITICAL: CRITICAL: integration.integration-castor03.diskspace._srv.byte_percentfree (<11.11%) [17:44:47] 10Release-Engineering-Team, 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10CDanis) 05Open→03Resolved Email with temporary Kerberos password sent. [17:46:50] 10Release-Engineering-Team, 10Platform Team, 10Platform Team Initiatives (API Gateway): Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 (10Pchelolo) [17:50:02] (03PS1) 10Ppchelko: Start branching OAuthRateLimiter extension for production [tools/release] - 10https://gerrit.wikimedia.org/r/614804 (https://phabricator.wikimedia.org/T258423) [18:08:05] thcipriani: ^ Forza has just been in #wikimedia saying they've got the verification email from phab and accepted it but it's not marking their account as verified [18:08:22] Thanks, RhinosF1 =) [18:08:45] I registered as Forza1000 [18:09:18] https://phabricator.wikimedia.org/p/Forza1000/ [18:11:14] 10Beta-Cluster-Infrastructure, 10Privacy Engineering, 10WMF-Legal, 10Privacy, and 2 others: Beta cluster: rules for permissions requiring confidentiality agreement - https://phabricator.wikimedia.org/T248546 (10sbassett) [18:11:59] Yea. When try to login to phabricator I get this: https://paste.tnonline.net/files/0rxyUTLfl7us_forza.png [18:14:10] Try it again, it can't hurt [18:14:22] Yep. I did [18:14:43] "Another verification email was sent to forza@....net" :/ [18:15:44] Perhaps it is simply waiting a little bit? :) [18:25:08] FYI: Manually cleaning up db after problematic block of `DannyS712 test2` - can't reblock, says already blocked, cannot unblock, says not blocked [18:28:04] 10Release-Engineering-Team, 10Platform Team, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 (10Pchelolo) [18:40:30] maintenance-disconnect-full-disks build 195883 integration-agent-docker-1011 (/srv: 96%): OFFLINE due to disk space [18:46:26] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<55.56%) [18:50:28] maintenance-disconnect-full-disks build 195885 integration-agent-docker-1011: OFFLINE due to disk space [18:56:27] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [18:57:07] 10Release-Engineering-Team-TODO, 10MediaWiki-Debug-Logger, 10Performance-Team, 10Platform Team, and 3 others: Ensure flood of hard-deprecations are caught during (train) deployments - https://phabricator.wikimedia.org/T252923 (10Krinkle) 05Open→03Resolved [19:15:54] maintenance-disconnect-full-disks build 195890 integration-agent-docker-1011: OFFLINE due to disk space [19:34:07] 10Release-Engineering-Team, 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10Halfak) 05Resolved→03Open Still waiting on deployment-prep access so that @calbon can do a beta deploy of ORES. [19:40:27] maintenance-disconnect-full-disks build 195895 integration-agent-docker-1011: OFFLINE due to disk space [19:44:53] Hey folks! Could someone help me get chrisalbon (calbon) deployment-prep access so he can do some ORES beta deploys? https://horizon.wikimedia.org/project/ [19:45:04] See https://phabricator.wikimedia.org/T256412 [19:50:10] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) I have confirmed that docker build will leave a container around if a build step fails. [19:54:36] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) From `docker build` docs: --force-rm=true|false Always remove intermediate... [19:55:25] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) a:03dancy [20:00:00] thcipriani, RhinosF1 It still doesn't work with the login to the phabricator :/ No confirmation emails are arriving, but I get mediawiki emails fine, and the signup email too. [20:01:11] (03PS1) 10Ahmon Dancy: Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) [20:01:50] (03CR) 10jerkins-bot: [V: 04-1] Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) (owner: 10Ahmon Dancy) [20:03:13] (03CR) 10DannyS712: [C: 03+1] "LGTM" [tools/release] - 10https://gerrit.wikimedia.org/r/614804 (https://phabricator.wikimedia.org/T258423) (owner: 10Ppchelko) [20:05:28] maintenance-disconnect-full-disks build 195900 integration-agent-docker-1011: OFFLINE due to disk space [20:06:52] 10Release-Engineering-Team, 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10thcipriani) [20:07:18] 10Release-Engineering-Team, 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10thcipriani) 05Open→03Resolved >>! In T256412#6320739, @Halfak wrote: > Still waiting on deployment-prep access so that... [20:30:28] maintenance-disconnect-full-disks build 195905 integration-agent-docker-1011: OFFLINE due to disk space [20:33:28] (03PS1) 10Jeena Huneidi: Fix UnsupportedOperationException for MultiBinding [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614855 (https://phabricator.wikimedia.org/T257526) [20:45:25] !log being testing changes to maintenance-disconnect-full-disks [20:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:50:28] (03CR) 10Thcipriani: [C: 03+1] "effective +2, will leave it to you to +2 (i.e., deploy)" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614855 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi) [20:53:10] (03CR) 10Jeena Huneidi: [C: 03+2] Fix UnsupportedOperationException for MultiBinding [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614855 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi) [20:53:44] (03Merged) 10jenkins-bot: Fix UnsupportedOperationException for MultiBinding [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614855 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi) [21:05:44] maintenance-disconnect-full-disks build 195921 integration-agent-docker-1002 (/srv: 100%): OFFLINE due to disk space [21:07:37] PROBLEM - Free space - all mounts on integration-agent-docker-1002 is CRITICAL: CRITICAL: integration.integration-agent-docker-1002.diskspace._srv.byte_percentfree (<20.00%) [21:11:02] maintenance-disconnect-full-disks build 195923 integration-agent-docker-1010 (/srv: 100%): OFFLINE due to disk space [21:11:42] PROBLEM - Free space - all mounts on integration-agent-docker-1010 is CRITICAL: CRITICAL: integration.integration-agent-docker-1010.diskspace._srv.byte_percentfree (<10.00%) [21:17:37] RECOVERY - Free space - all mounts on integration-agent-docker-1002 is OK: OK: All targets OK [21:18:44] PROBLEM - Free space - all mounts on integration-agent-docker-1012 is CRITICAL: CRITICAL: integration.integration-agent-docker-1012.diskspace._srv.byte_percentfree (<10.00%) [21:21:15] maintenance-disconnect-full-disks build 195925 integration-agent-docker-1002: OFFLINE due to disk space [21:21:15] maintenance-disconnect-full-disks build 195925 integration-agent-docker-1010: OFFLINE due to disk space [21:21:16] maintenance-disconnect-full-disks build 195925 integration-agent-docker-1011: OFFLINE due to disk space [21:21:39] RECOVERY - Free space - all mounts on integration-agent-docker-1010 is OK: OK: All targets OK [21:26:41] 10Release-Engineering-Team, 10Security-Team, 10user-sbassett: php-composer-security-docker failing due to git fetch of non-existent REL1_35 ref - https://phabricator.wikimedia.org/T257080 (10sbassett) 05Stalled→03Resolved Should be fixed now. [21:26:45] PROBLEM - Free space - all mounts on integration-agent-docker-1003 is CRITICAL: CRITICAL: integration.integration-agent-docker-1003.diskspace._srv.byte_percentfree (<11.11%) [21:28:43] RECOVERY - Free space - all mounts on integration-agent-docker-1012 is OK: OK: All targets OK [21:35:07] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins, 10User-brennen: maintenance-disconnect-full-disks: Agent nodes seem to be accumulating workspa... - https://phabricator.wikimedia.org/T258448 [21:35:55] (03PS1) 10Brennen Bearnes: WIP: maintenance-disconnect-full-disks: remove old workspaces [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) [21:36:46] RECOVERY - Free space - all mounts on integration-agent-docker-1003 is OK: OK: All targets OK [21:37:08] (03CR) 10Brennen Bearnes: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) (owner: 10Brennen Bearnes) [21:40:28] maintenance-disconnect-full-disks build 195930 integration-agent-docker-1002: OFFLINE due to disk space [21:40:29] maintenance-disconnect-full-disks build 195930 integration-agent-docker-1010: OFFLINE due to disk space [21:40:29] maintenance-disconnect-full-disks build 195930 integration-agent-docker-1011: OFFLINE due to disk space [21:41:46] (03PS2) 10Ahmon Dancy: Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) [21:44:11] PROBLEM - Free space - all mounts on integration-agent-docker-1007 is CRITICAL: CRITICAL: integration.integration-agent-docker-1007.diskspace._srv.byte_percentfree (<22.22%) [21:50:21] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<22.22%) [21:54:12] RECOVERY - Free space - all mounts on integration-agent-docker-1007 is OK: OK: All targets OK [21:58:16] woah [21:58:21] why is everything filling up [22:00:24] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [22:00:55] !log onlined integration-agent-docker-1002 [22:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:01:23] !log onlined integration-agent-docker-1010 [22:01:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:02:14] !log onlined integration-agent-docker-1011 [22:02:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:02:33] 10Beta-Cluster-Infrastructure: deployment-puppetmaster04: git-sync-upstream is failing with a merge conflict since 2020-07-17T08:50:01Z - https://phabricator.wikimedia.org/T258451 (10Mholloway) [22:02:50] 10Beta-Cluster-Infrastructure: deployment-puppetmaster04: git-sync-upstream is failing with a merge conflict since 2020-07-17T08:50:01Z - https://phabricator.wikimedia.org/T258451 (10Mholloway) p:05Triage→03High [22:03:41] 10Beta-Cluster-Infrastructure: deployment-puppetmaster04: git-sync-upstream is failing with a merge conflict since 2020-07-17T08:50:01Z - https://phabricator.wikimedia.org/T258451 (10Mholloway) [22:17:30] !log offlining integration-agent-docker-1003 for maintenance-disconnect-full-disks testing [22:17:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:28:35] maintenance-disconnect-full-disks build 195940 integration-agent-docker-1003 (/: 17%): OFFLINE due to disk space [22:28:36] maintenance-disconnect-full-disks build 195940 integration-agent-docker-1003 (/srv: 6%): still OFFLINE due to disk space [22:28:37] maintenance-disconnect-full-disks build 195940 integration-agent-docker-1003 (/var/lib/docker: 4%): RECOVERY disk space OK [22:29:10] ^ some of these are in error [22:52:40] (03PS1) 10Brennen Bearnes: DNM: maintenance-disconnect-full-disks: bring fixed nodes online [integration/config] - 10https://gerrit.wikimedia.org/r/614899 [22:58:16] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Patch-For-Review: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) I ran this today : ` dancy@contint1001:~$ docker container prune WARNI... [23:08:48] brennen: having it auto-online nodes sounds amazing :D [23:09:32] legoktm: yeah, soon. :) [23:32:29] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [23:42:28] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [23:54:19] PROBLEM - Host deployment-sentry01 is DOWN: CRITICAL - Host Unreachable (172.16.5.16)