[01:01:33] 10Continuous-Integration-Infrastructure, 10phpunit-patch-coverage: doc.wikimedia.org CSP prevents inline styles on HTML patch coverage report from working - https://phabricator.wikimedia.org/T215115 (10Bawolff) Which url is this? AFAIK we haven't enabled CSP on doc.wikimedia.org yet (T213223). So unless unsaf... [01:22:06] 10Continuous-Integration-Infrastructure, 10phpunit-patch-coverage: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10Krinkle) [01:23:59] 10Continuous-Integration-Infrastructure, 10phpunit-patch-coverage: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10Krinkle) These are not documentation pages published to doc.wikimedia.org post-merge (trusted). Instead,... [01:29:28] 10Continuous-Integration-Infrastructure, 10phpunit-patch-coverage: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10Legoktm) Sorry about the domain confusion. It was supposed to be integration.wikimedia.org. >>! In T21511... [01:29:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10phpunit-patch-coverage: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10Krinkle) [09:00:59] 10MediaWiki-Codesniffer: Enforce @covers… tags to have full qualified class names starting with backslash - https://phabricator.wikimedia.org/T215144 (10thiemowmde) [09:03:42] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "As far as I'm aware of, the critical prerequisite for this is T205063, which is already resolved." [tools/release] - 10https://gerrit.wikimedia.org/r/487623 (https://phabricator.wikimedia.org/T208499) (owner: 10Legoktm) [09:24:12] (03CR) 10Addshore: [C: 03+1] Stop deploying WikibaseQuality [tools/release] - 10https://gerrit.wikimedia.org/r/487623 (https://phabricator.wikimedia.org/T208499) (owner: 10Legoktm) [10:26:38] 10Gerrit, 10Phabricator: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10hashar) [11:14:30] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata Query UI, 10Jenkins, 10Patch-For-Review: wikidata/query/gui CI job lacks PhantomJS / proper browsers - https://phabricator.wikimedia.org/T183831 (10Addshore) [11:17:34] (03CR) 10Hashar: "My bad. I am terrible at code reviewing :(" [integration/config] - 10https://gerrit.wikimedia.org/r/487376 (owner: 10Kosta Harlan) [11:17:57] (03CR) 10Hashar: [C: 03+2] Add MediaWiki extension WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/487564 (owner: 10Mholloway) [11:19:57] (03Merged) 10jenkins-bot: Add MediaWiki extension WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/487564 (owner: 10Mholloway) [11:20:31] 10Diffusion, 10Gerrit: rPHAB to gerrit mirror stuck at 447032e1d - https://phabricator.wikimedia.org/T213512 (10MarcoAurelio) @mmodell Could you please take a look? Thanks. [11:21:38] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864 (10MarcoAurelio) Sorry for the off-topic. Maybe we should move towards using 2FA for gerrit instead, and require it for people with +2 access on `mediawiki` as well as those on `ldap/{wmf,ops}`, `operations/pu... [11:26:10] (03PS1) 10Hashar: fabfile: simplify a string concatenation [integration/config] - 10https://gerrit.wikimedia.org/r/487822 [11:27:40] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] Stop deploying WikibaseQuality [tools/release] - 10https://gerrit.wikimedia.org/r/487623 (https://phabricator.wikimedia.org/T208499) (owner: 10Legoktm) [11:34:12] (03PS1) 10Hashar: Update flake8 3.5.0 to 3.7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/487824 [11:37:43] (03CR) 10Hashar: [C: 03+2] Stop deploying WikibaseQuality [tools/release] - 10https://gerrit.wikimedia.org/r/487623 (https://phabricator.wikimedia.org/T208499) (owner: 10Legoktm) [11:38:17] (03Merged) 10jenkins-bot: Stop deploying WikibaseQuality [tools/release] - 10https://gerrit.wikimedia.org/r/487623 (https://phabricator.wikimedia.org/T208499) (owner: 10Legoktm) [11:38:33] 10Release-Engineering-Team (Backlog), 10Wikidata, 10Patch-For-Review: Stop branching & deploying WikibaseQuality extension - https://phabricator.wikimedia.org/T208499 (10hashar) WikibaseQuality should thus not be cut on Feb 5th when `1.33.0-wmf.16` is cut. The deploy task is T206670 [11:40:38] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T206670 (10hashar) Note: the WikibaseQuality MediaWiki extension will not be cut/included ( T208499 ). The extension has been merged in another T205063 and it is no more... [12:01:03] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:05:29] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:07:33] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10hashar) Icinga does not monitor Gerrit CPU usage / system load. We would need to add the `check_load` plugin mentioned above by @Dzahn. [12:11:37] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:15:59] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:27:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: extensions phpunit tests time out - https://phabricator.wikimedia.org/T214978 (10hashar) [12:45:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: extensions phpunit tests time out - https://phabricator.wikimedia.org/T214978 (10hashar) The patch was PS25 of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseCirrusSearch/+/447954/ the job timed out afte... [12:45:14] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864 (10Paladox) Gerrit dosent support 2fa as far as I know. [15:18:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [15:21:16] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [15:25:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [15:25:43] !log removed Jenkins user "nodepoolmanager" as well as related authorizations | T209361 [15:25:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:25:46] T209361: Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 [15:29:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10serviceops: Remove graphite data for nodepool - https://phabricator.wikimedia.org/T215172 (10hashar) p:05Triage→03Normal [15:30:14] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10hashar) Thank you @Marostegui ! [15:30:59] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [15:33:30] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Clean labnodepool1001.eqiad.wmnet from firewall / router - https://phabricator.wikimedia.org/T215173 (10hashar) p:05Triage→03Normal [15:33:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [15:35:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) I have filled sub tasks for the other teams to act on :) [15:35:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) 05Open→03Stalled [15:36:03] 10Continuous-Integration-Infrastructure (shipyard), 10cloud-services-team (Kanban): Clean labnodepool1001.eqiad.wmnet from firewall / router - https://phabricator.wikimedia.org/T215173 (10hashar) a:05hashar→03None [15:36:19] 10Continuous-Integration-Infrastructure (shipyard), 10serviceops: Remove graphite data for nodepool - https://phabricator.wikimedia.org/T215172 (10hashar) [16:33:24] 10Continuous-Integration-Infrastructure (shipyard), 10serviceops: Remove graphite data for nodepool - https://phabricator.wikimedia.org/T215172 (10greg) Doesn't the data just fall out after a while? [16:35:16] maintenance-disconnect-full-disks build 43817 integration-slave-docker-1040 (/var/lib/docker: 96%): OFFLINE due to disk space [16:35:17] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864 (10bd808) >>! In T147864#4924028, @MarcoAurelio wrote: > Does Gerrit support 2FA? Currently the two-factor protection for Wikimedia developer accounts must be handled by the Wikitech MediaWiki deployment. The... [16:36:14] (03PS1) 10Kosta Harlan: Sonar: Specify branch name and target [integration/config] - 10https://gerrit.wikimedia.org/r/487877 (https://phabricator.wikimedia.org/T215175) [16:41:09] (03PS1) 10Kosta Harlan: Sonar: Enable experimental for core, skins, and extensions [integration/config] - 10https://gerrit.wikimedia.org/r/487880 (https://phabricator.wikimedia.org/T215177) [16:42:22] (03CR) 10Kosta Harlan: sonar: run sonar analysis as a pre-merge step (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/487786 (https://phabricator.wikimedia.org/T215135) (owner: 10Gehel) [16:47:25] 10Phabricator, 10Release-Engineering-Team, 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10EBernhardson) This upgrade should be happening late febuary or early march. [16:50:14] maintenance-disconnect-full-disks build 43820 integration-slave-docker-1040: OFFLINE due to disk space [17:04:51] I believe this https://phabricator.wikimedia.org/T64053 will be possible now as we can wrap gr-tooltip around any text that starts with T[0-9] [17:15:13] maintenance-disconnect-full-disks build 43825 integration-slave-docker-1040: OFFLINE due to disk space [17:19:51] 10Gerrit, 10Upstream: Gerrit should feature customizable message on Login page (No 'Forgot password' link in the gerrit login page.) - https://phabricator.wikimedia.org/T60205 (10Paladox) We can use js to do this :) [17:36:41] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Brennen - https://phabricator.wikimedia.org/T214556 (10greg) a:03brennen [17:37:35] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Brennen - https://phabricator.wikimedia.org/T214556 (10greg) [17:40:18] maintenance-disconnect-full-disks build 43830 integration-slave-docker-1040: OFFLINE due to disk space [17:55:23] (03PS1) 10Hashar: Update Qiubble containers to npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) [17:55:52] (03CR) 10Hashar: [C: 04-1] "Untested. Probably want to rebuild them first to catch with updates then do this change." [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [17:56:22] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Brennen - https://phabricator.wikimedia.org/T214556 (10brennen) [17:57:51] 10Phabricator, 10Release-Engineering-Team, 10DBA, 10serviceops, and 2 others: Improve privilege separation for phabricator's config files and mysql credentials - https://phabricator.wikimedia.org/T146055 (10Dzahn) [17:57:54] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10CDanis) If the thing we want to monitor is "Gerrit is responding slowly / not at all", IMO that is the thing we should check. High CPU load is just... [18:05:21] maintenance-disconnect-full-disks build 43835 integration-slave-docker-1040: OFFLINE due to disk space [18:05:48] (03CR) 10jerkins-bot: [V: 04-1] Update Qiubble containers to npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [18:07:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: extensions phpunit tests time out - https://phabricator.wikimedia.org/T214978 (10Smalyshev) > Hence at the end of mw-debug-cli.log we have: Ah, that solves it I guess. Thanks! Though I wish there was an easier way to do... [18:08:37] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Dzahn) >>! In T215033#4925098, @CDanis wrote: > Would it be difficult to use `check_http`, .. pointed at a few key Gerrit URLs? We already check i... [18:10:18] maintenance-disconnect-full-disks build 43836 integration-slave-docker-1038 (/var/lib/docker: 97%): OFFLINE due to disk space [18:30:14] maintenance-disconnect-full-disks build 43840 integration-slave-docker-1038: OFFLINE due to disk space [18:30:15] maintenance-disconnect-full-disks build 43840 integration-slave-docker-1040: OFFLINE due to disk space [18:30:24] (03CR) 10Jforrester: "Don't we want to extend the node10 ones?" [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [18:42:58] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10CDanis) The timeouts on that check_ssl invocation make little sense to me -- a warning after 60 seconds, but critical after 30?... [18:46:26] (03PS2) 10Gehel: sonar: run sonar analysis as a pre-merge step [integration/config] - 10https://gerrit.wikimedia.org/r/487786 (https://phabricator.wikimedia.org/T215135) [18:47:32] (03CR) 10jerkins-bot: [V: 04-1] sonar: run sonar analysis as a pre-merge step [integration/config] - 10https://gerrit.wikimedia.org/r/487786 (https://phabricator.wikimedia.org/T215135) (owner: 10Gehel) [18:48:09] (03CR) 10Gehel: sonar: run sonar analysis as a pre-merge step (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/487786 (https://phabricator.wikimedia.org/T215135) (owner: 10Gehel) [18:50:59] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Dzahn) >>! In T215033#4925229, @CDanis wrote: > a warning after 60 seconds, but critical after 30? Those seem backwards Agre... [18:54:19] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Paladox) Im thinking we need the health check plugin. That way we just need to check the http status code. [18:55:16] maintenance-disconnect-full-disks build 43845 integration-slave-docker-1038: OFFLINE due to disk space [18:55:16] maintenance-disconnect-full-disks build 43845 integration-slave-docker-1040: OFFLINE due to disk space [19:03:49] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Dzahn) >>! In T215033#4925229, @CDanis wrote: > The timeouts on that check_ssl invocation make little sense to me Actually i... [19:13:22] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Dzahn) P.S. The --warning and --critical values are not backwards because they are "days until expiry". [19:18:37] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) [19:19:45] 10Gerrit, 10Icinga, 10Operations, 10monitoring, 10Patch-For-Review: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) Technically "investigate why it did not alert" has been resolved. But of course we also... [19:20:13] maintenance-disconnect-full-disks build 43850 integration-slave-docker-1038: OFFLINE due to disk space [19:20:13] maintenance-disconnect-full-disks build 43850 integration-slave-docker-1040: OFFLINE due to disk space [19:20:20] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) [19:45:15] maintenance-disconnect-full-disks build 43855 integration-slave-docker-1038: OFFLINE due to disk space [19:45:16] maintenance-disconnect-full-disks build 43855 integration-slave-docker-1040: OFFLINE due to disk space [19:46:23] (03CR) 10Krinkle: "Yeah, James/Lego and I agreed we'd migrate straight from npm (= node6 + npm3) to node10 (= node10 + npm6) - skipping the "npm6" phase (= n" [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [19:55:14] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Paladox) So we just need a http check that checks the website (without checking if the ssl cert is val... [19:57:44] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10CDanis) >>! In T215033#4925462, @Paladox wrote: > So we just need a http check that checks the website... [20:10:13] maintenance-disconnect-full-disks build 43860 integration-slave-docker-1038: OFFLINE due to disk space [20:10:14] maintenance-disconnect-full-disks build 43860 integration-slave-docker-1040: OFFLINE due to disk space [20:10:23] 10Continuous-Integration-Infrastructure (shipyard), 10cloud-services-team (Kanban): Clean labnodepool1001.eqiad.wmnet from firewall / router - https://phabricator.wikimedia.org/T215173 (10Andrew) a:03ayounsi Assigning to @ayounsi, hoping he can check the routers for this. There are no more references in pup... [20:10:35] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) Indeed, i would say merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/487901 should... [20:12:01] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) >>! In T215033#4925518, @Dzahn wrote: > healthcheck plugin you mentioned. Maybe in a separate t... [20:12:16] 10Gerrit, 10Icinga, 10Operations, 10monitoring, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Paladox) Already have T214326 for the health check plugin :) [20:16:18] Krenair or thcipriani, the first item in T210993 refers to 'role::beta::availability_collector' which is still applied on a single VM but hasn't been updated in years. Does anyone want to take on refactoring that to use prometheus, and/or removing it entirely? [20:16:19] T210993: Deprecate Diamond collectors in Cloud VPS - https://phabricator.wikimedia.org/T210993 [20:16:36] 10Continuous-Integration-Infrastructure (shipyard), 10cloud-services-team (Kanban): Clean labnodepool1001.eqiad.wmnet from firewall / router - https://phabricator.wikimedia.org/T215173 (10ayounsi) a:05ayounsi→03hashar No mention of that IP in Rancid (router configs). [20:16:47] I don't know much about prometheus [20:18:19] do you know what an 'availability collector' is? [20:20:08] andrewbogott: a custom "plugin" for diamond that received the status of things from varnish [20:20:11] "a custom collector deployed via Puppet (modules/diamond/files/collector/varnishstatus.py" [20:21:51] it seems about varnish in beta .. deployment-cache-upload [20:23:30] collects data if beta is up.. runs varnishtop [20:25:22] yeah, I'm wondering if the metric is read by anything/anyone [20:25:27] or if we can rip it out [20:32:14] if anything it might be read by a shinken check? [20:32:17] otherwise no idea [20:35:13] maintenance-disconnect-full-disks build 43865 integration-slave-docker-1038: OFFLINE due to disk space [20:35:13] maintenance-disconnect-full-disks build 43865 integration-slave-docker-1040: OFFLINE due to disk space [21:01:20] maintenance-disconnect-full-disks build 43870 integration-slave-docker-1038: OFFLINE due to disk space [21:01:21] maintenance-disconnect-full-disks build 43870 integration-slave-docker-1040: OFFLINE due to disk space [21:01:25] legoktm when will you be able to push the code you were talking about earlier? :) [21:22:29] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th - https://phabricator.wikimedia.org/T215204 (10EBernhardson) [21:25:14] maintenance-disconnect-full-disks build 43875 integration-slave-docker-1038: OFFLINE due to disk space [21:25:15] maintenance-disconnect-full-disks build 43875 integration-slave-docker-1040: OFFLINE due to disk space [21:50:12] maintenance-disconnect-full-disks build 43880 integration-slave-docker-1038: OFFLINE due to disk space [21:50:13] maintenance-disconnect-full-disks build 43880 integration-slave-docker-1040: OFFLINE due to disk space [21:51:20] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th - https://phabricator.wikimedia.org/T215204 (10EBernhardson) 05Open→03Resolved a:03EBernhardson I restarted logstash on `deployment-logstash2.deployment-prep.eqiad.wmflabs` and... [22:15:14] maintenance-disconnect-full-disks build 43885 integration-slave-docker-1038: OFFLINE due to disk space [22:15:14] maintenance-disconnect-full-disks build 43885 integration-slave-docker-1040: OFFLINE due to disk space [22:40:14] maintenance-disconnect-full-disks build 43890 integration-slave-docker-1038: OFFLINE due to disk space [22:40:15] maintenance-disconnect-full-disks build 43890 integration-slave-docker-1040: OFFLINE due to disk space [23:05:14] maintenance-disconnect-full-disks build 43895 integration-slave-docker-1038: OFFLINE due to disk space [23:05:15] maintenance-disconnect-full-disks build 43895 integration-slave-docker-1040: OFFLINE due to disk space [23:12:47] !log integration-slave-docker-1038:sudo docker image prune and bring back online [23:12:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:13:56] !log integration-slave-docker-1040:sudo docker image prune and bring back online [23:13:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:14:10] * thcipriani goes back to vacation day [23:35:58] based on hashar feedback i've done https://gerrit-review.googlesource.com/c/plugins/zuul-status/+/212792 [23:36:05] which should keep everyone happy :) [23:57:54] 10Release-Engineering-Team (Watching / External), 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10Krinkle)