[00:07:24] PROBLEM - Host deployment-cumin is DOWN: CRITICAL - Host Unreachable (172.16.5.1) [00:10:38] PROBLEM - Host deployment-cache-upload04 is DOWN: CRITICAL - Host Unreachable (172.16.5.14) [00:20:12] maintenance-disconnect-full-disks build 63780 integration-slave-docker-1037: OFFLINE due to disk space [00:20:13] maintenance-disconnect-full-disks build 63780 integration-slave-docker-1050: OFFLINE due to disk space [00:45:12] maintenance-disconnect-full-disks build 63785 integration-slave-docker-1037: OFFLINE due to disk space [00:45:12] maintenance-disconnect-full-disks build 63785 integration-slave-docker-1050: OFFLINE due to disk space [01:10:12] maintenance-disconnect-full-disks build 63790 integration-slave-docker-1037: OFFLINE due to disk space [01:10:13] maintenance-disconnect-full-disks build 63790 integration-slave-docker-1050: OFFLINE due to disk space [01:35:12] maintenance-disconnect-full-disks build 63795 integration-slave-docker-1037: OFFLINE due to disk space [01:35:12] maintenance-disconnect-full-disks build 63795 integration-slave-docker-1050: OFFLINE due to disk space [02:00:13] maintenance-disconnect-full-disks build 63800 integration-slave-docker-1037: OFFLINE due to disk space [02:00:13] maintenance-disconnect-full-disks build 63800 integration-slave-docker-1050: OFFLINE due to disk space [02:25:12] maintenance-disconnect-full-disks build 63805 integration-slave-docker-1037: OFFLINE due to disk space [02:25:12] maintenance-disconnect-full-disks build 63805 integration-slave-docker-1050: OFFLINE due to disk space [02:50:14] maintenance-disconnect-full-disks build 63810 integration-slave-docker-1037: OFFLINE due to disk space [02:50:15] maintenance-disconnect-full-disks build 63810 integration-slave-docker-1050: OFFLINE due to disk space [03:15:13] maintenance-disconnect-full-disks build 63815 integration-slave-docker-1037: OFFLINE due to disk space [03:15:13] maintenance-disconnect-full-disks build 63815 integration-slave-docker-1050: OFFLINE due to disk space [03:40:13] maintenance-disconnect-full-disks build 63820 integration-slave-docker-1037: OFFLINE due to disk space [03:40:13] maintenance-disconnect-full-disks build 63820 integration-slave-docker-1050: OFFLINE due to disk space [04:05:12] maintenance-disconnect-full-disks build 63825 integration-slave-docker-1037: OFFLINE due to disk space [04:05:12] maintenance-disconnect-full-disks build 63825 integration-slave-docker-1050: OFFLINE due to disk space [04:30:12] maintenance-disconnect-full-disks build 63830 integration-slave-docker-1037: OFFLINE due to disk space [04:30:12] maintenance-disconnect-full-disks build 63830 integration-slave-docker-1050: OFFLINE due to disk space [04:55:13] maintenance-disconnect-full-disks build 63835 integration-slave-docker-1037: OFFLINE due to disk space [04:55:13] maintenance-disconnect-full-disks build 63835 integration-slave-docker-1050: OFFLINE due to disk space [05:20:12] maintenance-disconnect-full-disks build 63840 integration-slave-docker-1037: OFFLINE due to disk space [05:20:12] maintenance-disconnect-full-disks build 63840 integration-slave-docker-1050: OFFLINE due to disk space [05:36:44] 10Scap, 10Operations, 10cloud-services-team: deploy1001 cannot reach cloudweb2001-dev.wikimedia.org when running scap - https://phabricator.wikimedia.org/T220931 (10Marostegui) [05:45:12] maintenance-disconnect-full-disks build 63845 integration-slave-docker-1037: OFFLINE due to disk space [05:45:12] maintenance-disconnect-full-disks build 63845 integration-slave-docker-1050: OFFLINE due to disk space [06:10:13] maintenance-disconnect-full-disks build 63850 integration-slave-docker-1037: OFFLINE due to disk space [06:10:13] maintenance-disconnect-full-disks build 63850 integration-slave-docker-1050: OFFLINE due to disk space [06:35:13] maintenance-disconnect-full-disks build 63855 integration-slave-docker-1037: OFFLINE due to disk space [06:35:13] maintenance-disconnect-full-disks build 63855 integration-slave-docker-1050: OFFLINE due to disk space [06:48:04] 10Gerrit, 10Code-Health, 10Test-Coverage, 10Upstream: Display per-line test coverage during code review - https://phabricator.wikimedia.org/T220938 (10Tgr) [06:57:39] 10Release-Engineering-Team, 10Epic: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294 (10Tgr) Some ideas I have been playing with: {T220939}, {T220938}. [07:00:12] maintenance-disconnect-full-disks build 63860 integration-slave-docker-1037: OFFLINE due to disk space [07:00:12] maintenance-disconnect-full-disks build 63860 integration-slave-docker-1050: OFFLINE due to disk space [07:25:12] maintenance-disconnect-full-disks build 63865 integration-slave-docker-1037: OFFLINE due to disk space [07:25:13] maintenance-disconnect-full-disks build 63865 integration-slave-docker-1050: OFFLINE due to disk space [07:50:12] maintenance-disconnect-full-disks build 63870 integration-slave-docker-1037: OFFLINE due to disk space [07:50:12] maintenance-disconnect-full-disks build 63870 integration-slave-docker-1050: OFFLINE due to disk space [08:13:22] <_joe_> hashar: quite a few VMs are out of disk space [08:26:07] _joe_: yes I noticed that. There is something somehow that is filling the /srv partition :-(((( [08:34:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) [08:38:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) The last few builds on that host: ` 2019-04-13 13:49:08,614 build:quibble-vendor-mysql-hhvm-docker {u'n... [08:48:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) What I have spotted is the job mwselenium-quibble-docker taking ages to synchronize a cache: ` 14:01:30... [08:51:17] !log castor: nuked /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwselenium-quibble-docker # T220948 [08:51:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:51:20] T220948: Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 [09:22:53] 10Gerrit, 10Release-Engineering-Team (Backlog), 10User-zeljkofilipin: PolyGerrit 2.15 UI does not support custom dashboards - https://phabricator.wikimedia.org/T220801 (10hashar) [09:42:06] 10Scap, 10Operations, 10cloud-services-team: deploy1001 cannot reach cloudweb2001-dev.wikimedia.org when running scap - https://phabricator.wikimedia.org/T220931 (10aborrero) By the time of this phab comment, cloudweb2001-dev.wikimedia.org is a spare system waiting to be put into service soon. We can safely... [09:42:48] 10Project-Admins, 10Developer-Advocacy, 10patch-welcome: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266 (10Aklapper) For the records, `#need-volunteer` got renamed to `#patch-welcome` by Krinkle in https://phabric... [09:54:48] 10Scap, 10Operations, 10cloud-services-team: deploy1001 cannot reach cloudweb2001-dev.wikimedia.org when running scap - https://phabricator.wikimedia.org/T220931 (10MoritzMuehlenhoff) JFTR; this needs to be dropped from hieradata/common/scap/dsh.yaml [10:16:08] !log ores:8f01d40 going beta [10:16:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:29:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) the synced cache directory is filled up by Chromium which honors `XDG_CACHE_HOME=/cache` which have in... [10:34:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) https://chromium.googlesource.com/chromium/src/+/HEAD/docs/user_data_dir.md On Linux, the user cache d... [10:38:47] 10Project-Admins: Add #SVG tag to Phabricator - https://phabricator.wikimedia.org/T194188 (10Aklapper) 05Open→03Declined Unfortunately closing this report as no further information has been provided. @Volker_E: After you have provided the information asked for and if this still happens, please set the statu... [11:36:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) The container has: | XDG_CACHE_HOME=/cache | XDG_CONFIG_HOME=/tmp Thus: 1 system config dir is `/tmp`... [12:01:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) The workaround is thus to move the system config dir to a subdirectory so that it is not the ancestor o... [12:06:48] (03PS1) 10Hashar: docker: prevent Chromium cache from being saved [integration/config] - 10https://gerrit.wikimedia.org/r/503973 (https://phabricator.wikimedia.org/T220948) [12:07:40] (03CR) 10Hashar: [C: 03+2] docker: prevent Chromium cache from being saved [integration/config] - 10https://gerrit.wikimedia.org/r/503973 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [12:09:38] (03Merged) 10jenkins-bot: docker: prevent Chromium cache from being saved [integration/config] - 10https://gerrit.wikimedia.org/r/503973 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [12:12:27] (03PS1) 10Hashar: Prevent Chromium to write its caches to XDG_CACHE_HOME [integration/config] - 10https://gerrit.wikimedia.org/r/503980 (https://phabricator.wikimedia.org/T220948) [12:15:22] (03CR) 10Hashar: [C: 03+2] Prevent Chromium to write its caches to XDG_CACHE_HOME [integration/config] - 10https://gerrit.wikimedia.org/r/503980 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [12:17:34] (03Merged) 10jenkins-bot: Prevent Chromium to write its caches to XDG_CACHE_HOME [integration/config] - 10https://gerrit.wikimedia.org/r/503980 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [12:22:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jenkins instances /srv suddenly becomes full causing it to be disconnected - https://phabricator.wikimedia.org/T220948 (10hashar) 05Open→03Resolved a:03hashar [13:11:39] (03CR) 10Hashar: [C: 03+2] [WebDAV] Switch to composer based quibble [integration/config] - 10https://gerrit.wikimedia.org/r/503727 (https://phabricator.wikimedia.org/T220888) (owner: 10Umherirrender) [13:13:06] (03Merged) 10jenkins-bot: [WebDAV] Switch to composer based quibble [integration/config] - 10https://gerrit.wikimedia.org/r/503727 (https://phabricator.wikimedia.org/T220888) (owner: 10Umherirrender) [13:34:37] 10Continuous-Integration-Config, 10MediaWiki-extensions-WebDAV, 10Patch-For-Review: WebDAV master is broken on PHP7.* CI: Use of undefined constant WEBDAV_AUTH_TOKEN - https://phabricator.wikimedia.org/T220888 (10hashar) 05Open→03Resolved a:03hashar Works now :) [13:36:03] 10Scap, 10Operations, 10cloud-services-team, 10Patch-For-Review: deploy1001 cannot reach cloudweb2001-dev.wikimedia.org when running scap - https://phabricator.wikimedia.org/T220931 (10Marostegui) 05Open→03Resolved a:03Marostegui After merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/50... [13:42:53] 10Continuous-Integration-Config: apc.enable_cli should be enabled in PHP jobs - https://phabricator.wikimedia.org/T220920 (10hashar) 05Open→03Declined In an extension test, I don't think there is any point in testing APCU. The integration testing should be done in MediaWiki core if ever done which would guan... [13:55:15] 10Continuous-Integration-Config: apc.enable_cli should be enabled in PHP jobs - https://phabricator.wikimedia.org/T220920 (10Daimona) @hashar That's fair, but how can we test that caching works for that bit of code, if the code is using APC to run tests? At this point, shouldn't the service be changed to return... [14:41:43] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2546 MB (5% inode=63%) [14:46:50] RECOVERY - Disk space on contint1001 is OK: DISK OK [14:55:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10fsero) it seems is almost full again. Did you consider to set up a periodic d... [14:56:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10Marostegui) Note that this task is about `/srv` and the current issue is with... [14:57:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10Marostegui) @fsero maybe comment at {T207702}? [14:58:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10fsero) oh sorry @Marostegui i misread :) [14:58:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10fsero) it seems is almost full again. Did you consider to set up a periodic docker image cleanup? cc @hashar a naive straightfo... [16:53:21] hashar: just an idea but maybe it would help to set xdg_cache to /tmp and then for specific things we want to cache (node_modules, composer, …?) configure those to use /cache. So by default it's temporary. [16:53:53] I imagine the number of things we want to cache and is secure, resistant, useful and self-cleaning is fairly small [16:57:08] looking at castor02, it seems mostly composer and npm. In some there are random other ones I don't recognise, not sure if intentional. [17:07:00] Krinkle: maybe. Previously the castor commands explicitly stated the list of directories to be saved [17:07:14] such as npm composer gems m2 (for maven) [17:07:34] I thought it would be simpler to just save everything ;D [17:07:39] fun thing [17:07:43] for some jobs, there is hundreds of directories in castor02. [17:07:43] win 59 [17:07:51] e.g. mwgate-npm-node-6-docker and selenium-daily-beta-MediaWiki [17:07:53] it only seem to happen for the mediawiki-selenium based job [17:07:55] looks like npm [17:08:02] maybe it's missing a sub directory for those jobs? [17:08:23] for all the others, they are in a sub directory "npm" [17:10:16] Krinkle: /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/npm ? [17:10:21] I would guess there is a branch "npm" somewhere [17:10:47] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/495287/ [17:10:48] yeah [17:11:26] we should probably rethinkg the way npm/composer stuff is cached for mediawiki repos anyway [17:26:55] (03PS4) 10Dduvall: pipeline: Directed graph execution model [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) [17:26:57] (03PS5) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [17:26:59] (03PS5) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [17:27:32] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [17:28:04] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [17:28:24] (03CR) 10Dduvall: pipeline: Directed graph execution model (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [17:28:40] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall) [17:33:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10hashar) We cant really do that since `docker-pkg` relies on those images being there. Though I have added support for it to auto... [17:34:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar) Since that is recurring. Can we check whether we can add a couple disks to the machine? I guess 25... [18:11:08] (03CR) 10Lars Wirzenius: [C: 03+1] "Spello" (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:12:12] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10Dzahn) @thcipriani Done. They have been added to the LDAP group. docs are: https://wikitech.wikimedia.org/wiki/L... [18:14:28] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10Dzahn) 05Open→03Resolved a:03Dzahn [18:15:15] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10greg) >>! In T219086#5112635, @Dzahn wrote: > In addition to roots.. there is a special admin group called `ldap-ad... [18:16:57] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10Dzahn) >>! In T219086#5112639, @greg wrote: > Should probably remove chad? Maybe add more people? (sorry, should be... [18:20:33] 10Gerrit, 10Release-Engineering-Team: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10thcipriani) [18:23:41] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10LarsWirzenius) Just to clarify, when you say GC, do you mean Java virtual machine garbage collection or git repository object garbage collection? [18:25:11] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10Paladox) @LarsWirzenius he's referring to JVM GC :) [18:25:59] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10thcipriani) >>! In T221026#5112672, @LarsWirzenius wrote: > Just to clarify, when you say GC, do you mean Java virtual machine garbage collection or git repository object ga... [19:56:44] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: scap no longer !log'ging to server admin log - https://phabricator.wikimedia.org/T221035 (10Marostegui) [19:58:09] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: Switch Fresnel for MediaWiki jobs from php70 to php72 - https://phabricator.wikimedia.org/T220866 (10Krinkle) p:05Triage→03Low [20:17:14] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: scap no longer !log'ging to server admin log - https://phabricator.wikimedia.org/T221035 (10Peachey88) [20:17:19] 10Continuous-Integration-Infrastructure: Fix partitions on CI slaves , some are missing /var/lib/docker - https://phabricator.wikimedia.org/T221040 (10Paladox) [20:17:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Fix partitions on CI slaves , some are missing /var/lib/docker - https://phabricator.wikimedia.org/T221040 (10hashar) That is set by profile::ci::dockervolume which comes from df1dc5957edce36b12909b3e8bd8cc1b8935fb44 by @dduvall [20:31:36] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10Paladox) p:05Triage→03High [21:02:48] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [3.0] [21:04:35] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [21:30:17] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:36:16] 10Phabricator: Overwrite all my messages - https://phabricator.wikimedia.org/T221046 (10Micru) [21:52:13] greg-g: Hope https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=1823435&oldid=1823434 is OK. [21:53:09] James_F: godspeed [21:53:26] Thanks. :-) [22:57:46] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [2.0] [23:23:07] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: scap no longer !log'ging to server admin log - https://phabricator.wikimedia.org/T221035 (10bd808) ` [23:18] < bd808> logmsgbot isn't here? [23:19] < bd808> it should be running from icinga.wikimedia.org I think based on a quick grep of ops/... [23:33:58] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: scap no longer !log'ging to server admin log - https://phabricator.wikimedia.org/T221035 (10CDanis) 05Open→03Resolved a:03CDanis looks like `logmsgbot` was happily chattering away in #wikimedia-overload because of some race condition (within I... [23:42:55] 10Release-Engineering-Team (Kanban): Share results of the Developer Satisfaction survey - https://phabricator.wikimedia.org/T209913 (10greg) [23:42:57] 10Release-Engineering-Team (Kanban), 10User-greg: Summarize Developer Satisfaction survey data - https://phabricator.wikimedia.org/T209914 (10greg) 05Open→03Resolved a:03jeena This was a redundant task [23:43:16] 10Release-Engineering-Team, 10Epic: Complete first round of the Release Engineering Developer Satisfaction survey - https://phabricator.wikimedia.org/T197635 (10greg) [23:43:18] 10Release-Engineering-Team (Kanban): Share results of the Developer Satisfaction survey - https://phabricator.wikimedia.org/T209913 (10greg) 05Open→03Resolved a:03jeena This was a redundant task, apologies. [23:43:36] 10Release-Engineering-Team, 10Epic: Complete first round of the Release Engineering Developer Satisfaction survey - https://phabricator.wikimedia.org/T197635 (10greg) 05Open→03Resolved a:03jeena This was a redundant task, apologies. [23:44:54] 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations, 10Stashbot: scap no longer !log'ging to server admin log - https://phabricator.wikimedia.org/T221035 (10greg)