[01:56:03] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Volker_E) p:05Triage>03High [03:03:13] 10Continuous-Integration-Infrastructure, 10Jenkins: Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 (10Legoktm) 05stalled>03Open This was actually released in xunit-2.0.0, so someone will need to look a bit closer to see what (if any) b... [04:55:55] 10Phabricator (Upstream), 10Pywikibot, 10Upstream: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Xqt) Seem this dependeds on the used browser. With Safari typing โ€žpywโ€œ I get Pybal first and Pywikibot second. [05:07:50] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T191070 (10Krinkle) [05:34:16] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Legoktm) I think we could upgrade contint1001 to use PHP 7.2 from apt.wikimedia.org...not sure if an upgrade for that to stretch... [05:46:03] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [06:22:40] <_joe_> !log cherry-picking 455154 (php-fpm installation) on deployment-prep [06:22:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:07:40] !log deployment-maps04 deployed [kartotherian/deploy@27062b4]: Specify WDQS endpoint in the service config (T205607) [07:07:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:07:45] T205607: Kartotherian should use discovery endpoint to connect to wikidata query service - https://phabricator.wikimedia.org/T205607 [07:10:08] maintenance-disconnect-full-disks build 7999 integration-slave-docker-1038 (/srv: 95%): OFFLINE due to disk space [07:13:36] (03CR) 10Legoktm: [C: 04-1] "I don't think we want to support the @tag(something) syntax for all tags, just PHPMD's @SuppressWarnings should be checking for that." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/463590 (https://phabricator.wikimedia.org/T205768) (owner: 10MarkAHershberger) [07:15:07] maintenance-disconnect-full-disks build 8000 integration-slave-docker-1038: OFFLINE due to disk space [07:15:49] PROBLEM - Free space - all mounts on integration-slave-docker-1038 is CRITICAL: CRITICAL: integration.integration-slave-docker-1038.diskspace._srv.byte_percentfree (<22.22%) [07:17:33] (03CR) 10Legoktm: [C: 032] "Sounds good. Thanks :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [07:18:27] <_joe_> Krenair: sorry I just gave the wrong git command and removed "[Live hack] Try to make prometheus stuff work" from deployment-puppetmaster03 [07:18:38] <_joe_> I'm trying to recover it :/ [07:19:20] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) So the plan is: * Ignore the extra dbs for s5 connection on multisource host (dbstore1002, labsdb1009/10/11) with: ```... [07:19:56] (03CR) 10Legoktm: [C: 04-1] "Code looks good, see inline comment." (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/463656 (https://phabricator.wikimedia.org/T205800) (owner: 10Umherirrender) [07:21:06] <_joe_> Krenair: recovered; git reflog is of great help in such situations [07:21:21] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Smalyshev) [07:40:07] maintenance-disconnect-full-disks build 8005 integration-slave-docker-1038: OFFLINE due to disk space [08:05:07] maintenance-disconnect-full-disks build 8010 integration-slave-docker-1038: OFFLINE due to disk space [08:08:04] _joe_, np I think that one needed updating on beta anyway, some progress has been made in gerrit by ottomata [08:13:55] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [08:30:08] maintenance-disconnect-full-disks build 8015 integration-slave-docker-1038: OFFLINE due to disk space [08:55:07] maintenance-disconnect-full-disks build 8020 integration-slave-docker-1038: OFFLINE due to disk space [09:20:09] maintenance-disconnect-full-disks build 8025 integration-slave-docker-1038: OFFLINE due to disk space [09:26:48] 10Beta-Cluster-Infrastructure, 10Parsoid, 10RESTBase, 10Services (watching): Parsoid misconfigured in Beta - https://phabricator.wikimedia.org/T206107 (10mobrovac) p:05Triage>03High [09:43:30] <_joe_> Krenair: what is the tag you add to patches cherry-picked in beta? [09:44:25] beta-picked [09:44:32] might have forgotten it from some [09:44:48] <_joe_> I just want to add it to the new patch I cherry-picked [09:45:07] maintenance-disconnect-full-disks build 8030 integration-slave-docker-1038: OFFLINE due to disk space [09:45:14] <_joe_> I thought it was actually a good idea [09:45:17] ok [09:45:32] <_joe_> I was wondering what tags could be used for [10:10:07] maintenance-disconnect-full-disks build 8035 integration-slave-docker-1038: OFFLINE due to disk space [10:35:06] maintenance-disconnect-full-disks build 8040 integration-slave-docker-1038: OFFLINE due to disk space [11:00:07] maintenance-disconnect-full-disks build 8045 integration-slave-docker-1038: OFFLINE due to disk space [11:25:07] maintenance-disconnect-full-disks build 8050 integration-slave-docker-1038: OFFLINE due to disk space [11:25:39] 10Gerrit, 10Developer-Advocacy, 10Upstream: Gitiles should add username in git clone links, triggers "Permission denied (publickey)" error - https://phabricator.wikimedia.org/T189182 (10Aklapper) [11:50:07] maintenance-disconnect-full-disks build 8055 integration-slave-docker-1038: OFFLINE due to disk space [12:15:07] maintenance-disconnect-full-disks build 8060 integration-slave-docker-1038: OFFLINE due to disk space [12:18:02] 10Project-Admins, 10Wikidata, 10wikiba.se: permit a phabricator page for the FactGrid project - https://phabricator.wikimedia.org/T193071 (10Aklapper) 05stalled>03Resolved a:05Olaf_Simons>03Aklapper Requested public project #FactGrid has been created: https://phabricator.wikimedia.org/project/view/36... [12:40:08] maintenance-disconnect-full-disks build 8065 integration-slave-docker-1038: OFFLINE due to disk space [13:05:07] maintenance-disconnect-full-disks build 8070 integration-slave-docker-1038: OFFLINE due to disk space [13:13:10] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Ottomata) There are two changes needed, including https://gerrit.wikimedia.org/r... [13:14:11] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10CCicalese_WMF) 05Resolved>03Open Unfortunately, this rule is not working exactly as I had hoped. If a task is on a milestone workboard of one of the two projects, it is not considered to be tagged with the pr... [13:25:10] 10Gerrit: Unable to change full name through contact settings page or command line - https://phabricator.wikimedia.org/T206129 (10Matthew-a-thompson) [13:28:42] 10Continuous-Integration-Infrastructure: Add pre-commit hook that does basic checks like php -l - https://phabricator.wikimedia.org/T201778 (10Simetrical) I started using the following pre-commit hook: ``` #!/bin/bash # Redirect output to stderr. exec 1>&2 PHP_FILES=$(git diff --cached --name-only --diff-filt... [13:30:07] maintenance-disconnect-full-disks build 8075 integration-slave-docker-1038: OFFLINE due to disk space [13:35:05] 10Gerrit: Unable to change full name through contact settings page or command line - https://phabricator.wikimedia.org/T206129 (10Paladox) You have to edit your name through wikitech, then in gerrit you click reload next to full name. [13:53:03] 10Release-Engineering-Team (Watching / External), 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Current), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10CCicalese_WMF) [13:53:26] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.7-1 - https://phabricator.wikimedia.org/T204383 (10mmodell) [13:54:37] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.7-1 - https://phabricator.wikimedia.org/T204383 (10mmodell) repurposing this task since it's still open. ping @fgiunchedi can you upload the latest scap when you have a chance? Related Pup... [13:55:08] maintenance-disconnect-full-disks build 8080 integration-slave-docker-1038: OFFLINE due to disk space [14:02:56] 10Deployments, 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Core Platform Team ( Code Health (TEC13)), and 3 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10CCicalese_WMF) [14:17:45] 10Continuous-Integration-Infrastructure, 10Wikimedia-General-or-Unknown, 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Current), and 2 others: Deprecate/obsolete $wgWikimediaJenkinsCI - https://phabricator.wikimedia.org/T200650 (10CCicalese_WMF) [14:20:07] maintenance-disconnect-full-disks build 8085 integration-slave-docker-1038: OFFLINE due to disk space [14:27:12] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10mobrovac) The error, as seen by RESTBase: ``` error: Config Request failure for \"https://en.wikipedia.beta.wmflabs.org/w/api.ph... [14:27:33] 10Beta-Cluster-Infrastructure, 10Parsoid, 10RESTBase, 10Services (watching): Parsoid misconfigured in Beta - https://phabricator.wikimedia.org/T206107 (10mobrovac) [14:27:36] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10mobrovac) [14:27:51] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Aklapper) @CCicalese_WMF: Probably not 'automagically' possible until T144041 is resolved. We could probably add subprojects/milestones manually/explicitly to those rules (but that'll become a bit cumbersome if y... [14:30:08] maintenance-disconnect-full-disks build 8087 integration-slave-docker-1040 (/var/lib/docker: 95%): OFFLINE due to disk space [14:34:03] PROBLEM - Free space - all mounts on integration-slave-docker-1040 is CRITICAL: CRITICAL: integration.integration-slave-docker-1040.diskspace._var_lib_docker.byte_percentfree (<40.00%) [14:37:00] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10CCicalese_WMF) Oh, bummer. Yeah, adding the milestones explicitly whenever they get created would be quite cumbersome. Could you please disable or delete the two rules you created for now. Hopefully this will be... [14:43:13] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Aklapper) 05Open>03stalled Both rules archived. Can be re-enabled again later. [14:43:21] 10Phabricator: Develop a custom herald condition for maniphest tasks: "projects include any subprojects of ___ " - https://phabricator.wikimedia.org/T144041 (10Aklapper) [14:43:24] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Aklapper) [14:45:06] maintenance-disconnect-full-disks build 8090 integration-slave-docker-1038: OFFLINE due to disk space [14:45:07] maintenance-disconnect-full-disks build 8090 integration-slave-docker-1040: OFFLINE due to disk space [14:45:07] maintenance-disconnect-full-disks build 8090 integration-slave-docker-1043 (/var/lib/docker: 97%): OFFLINE due to disk space [14:48:23] PROBLEM - Free space - all mounts on integration-slave-docker-1043 is CRITICAL: CRITICAL: integration.integration-slave-docker-1043.diskspace._var_lib_docker.byte_percentfree (<40.00%) [14:53:18] !log integration-slave-docker-1038 removed workspaces, brought back online integration-slave-docker-104{0,3} need more investigation since docker is full(?) [14:53:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:55:51] (03PS1) 10Thcipriani: Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 [14:58:40] (03CR) 10jerkins-bot: [V: 04-1] Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 (owner: 10Thcipriani) [15:00:48] RECOVERY - Free space - all mounts on integration-slave-docker-1038 is OK: OK: All targets OK [15:02:09] (03PS2) 10Thcipriani: Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 [15:04:54] (03CR) 10jerkins-bot: [V: 04-1] Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 (owner: 10Thcipriani) [15:06:46] (03PS3) 10Thcipriani: Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 [15:09:45] (03CR) 10Thcipriani: [C: 032] Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 (owner: 10Thcipriani) [15:10:06] maintenance-disconnect-full-disks build 8095 integration-slave-docker-1040: OFFLINE due to disk space [15:10:07] maintenance-disconnect-full-disks build 8095 integration-slave-docker-1043: OFFLINE due to disk space [15:12:00] (03Merged) 10jenkins-bot: Use a separate gate-and-submit pipeline for l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/464159 (owner: 10Thcipriani) [15:12:53] !log reloading zuul to deploy l10 pipeline [15:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T191070 (10Krinkle) [15:31:02] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Nodes taken offline due to full /var/lib/docker partition - https://phabricator.wikimedia.org/T206134 (10dduvall) p:05Triage>03High [15:35:08] maintenance-disconnect-full-disks build 8100 integration-slave-docker-1040: OFFLINE due to disk space [15:35:09] maintenance-disconnect-full-disks build 8100 integration-slave-docker-1043: OFFLINE due to disk space [15:39:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Nodes taken offline due to full /var/lib/docker partition - https://phabricator.wikimedia.org/T206134 (10dduvall) It appears that Docker's JSON log files from some long running Quibble containers ar... [15:40:45] !log killing long running docker jobs on integration-slave-docker-1040/1043 which are filling up /var/lib/docker with log output [15:40:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:19] !log bringing integration-slave-docker-1040/1043 nodes back online after killing long running docker job and freeing up /var/lib/docker space (T206134) [15:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:23] T206134: Nodes taken offline due to full /var/lib/docker partition - https://phabricator.wikimedia.org/T206134 [15:52:57] 10Release-Engineering-Team (Watching / External), 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Later): Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in t... - https://phabricator.wikimedia.org/T106388 [15:53:12] 10Release-Engineering-Team (Watching / External), 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Later): Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a refer... - https://phabricator.wikimedia.org/T106388 [15:53:21] RECOVERY - Free space - all mounts on integration-slave-docker-1043 is OK: OK: All targets OK [15:53:47] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team ( Code Health (TEC13)), and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10CCicalese_WMF) [15:57:05] 10Continuous-Integration-Infrastructure, 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Later), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10CCicalese_WMF) [15:59:02] RECOVERY - Free space - all mounts on integration-slave-docker-1040 is OK: OK: All targets OK [16:18:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [16:21:10] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Arlolra) I'm assuming it's this series of patches that broke it https://github.com/wikimedia/puppet/commit/2f55fc2396c92c1a2d04... [16:49:30] (03PS1) 10Thcipriani: Change gate-and-submit-l10n to low precedence [integration/config] - 10https://gerrit.wikimedia.org/r/464184 [16:49:44] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Pchelolo) 09:37 Pchelolo: arlolra: I think I found why parsoid is failing in beta 09:39 arlolra: Pchelolo: /me perks up ... I hav... [16:49:51] (03CR) 10Thcipriani: [C: 032] Change gate-and-submit-l10n to low precedence [integration/config] - 10https://gerrit.wikimedia.org/r/464184 (owner: 10Thcipriani) [16:51:25] (03Merged) 10jenkins-bot: Change gate-and-submit-l10n to low precedence [integration/config] - 10https://gerrit.wikimedia.org/r/464184 (owner: 10Thcipriani) [16:52:02] !log reloading zuul to deploy Change gate-and-submit-l10n to low precedence [16:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:18:14] hi rel-eng! [17:18:33] Is the i18n HTML detection new? [17:18:35] https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/86/console [17:19:00] There are a bunch of DonationInterface messages with embedded links, etc. [17:19:14] Will they all need manual review now? [17:20:08] maintenance-disconnect-full-disks build 8121 integration-slave-docker-1034 (/srv: 95%): OFFLINE due to disk space [17:24:03] also, we're still seeing out of memory errors on CiviCRM tests [17:24:21] e.g. PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 32 bytes) in .... [17:24:34] generally trying to do database things [17:24:45] that one was on integration-slave-1004 [17:25:29] I'm pretty sure we cleared out the ibdata file on that yesterday [17:26:21] would it be possible to increase the size of /var/lib/mysql on the integration boxes? [17:34:29] or at least have a daily job to stop mysql / drop dbs / delete ibdata1 ? [17:38:53] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Ottomata) FINALLY GOT IT! https://beta-prometheus.wmflabs.org/beta/graph?g0.ran... [17:39:06] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Ottomata) a:03Ottomata [17:39:31] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Ottomata) [17:40:09] maintenance-disconnect-full-disks build 8125 integration-slave-docker-1034: OFFLINE due to disk space [17:42:01] ^ investigating [17:46:42] !log removing old workspaces on integration-slave-docker-1034 (`rm -rf /srv/jenkins-workspace/workspace/*`) and bringing back online [17:46:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:48:45] (03CR) 10Thcipriani: [C: 031] "Looks like it should do what we want." [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [17:48:50] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) Thats awesome!!!!๐ŸŽ‰ ๐ŸŽ‰ ๐ŸŽ‰ ๐ŸŽ‰ ๐ŸŽ‰ ๐ŸŽ‰ @Thank you @Ottomata @Krenair and @fgiu... [17:58:42] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Patch-For-Review, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Joe) While @Arlolra patch surely does a sensible thing, I am not sure we have an https interface to mediawi... [18:05:01] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Niedzielski) {icon thumbs-up}{icon thumbs-up}{icon thumbs-up}{icon thumbs-up}{ic... [18:05:54] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Krenair) so this is resolved? [18:10:10] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Patch-For-Review, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Joe) Sorry, I misinterpreted the issue. We are connecting via http even though the configuration says https... [18:12:27] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Patch-For-Review, 10Services (blocked): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Arlolra) 05Open>03Resolved a:03Arlolra [18:14:57] (03PS3) 10Dduvall: Wipe entire workspace after builds of Docker based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) [18:16:55] !log deploying I0fcd95427a6468213feb0be2ab7ab52f0d719f90 for 12 mediawiki-quibble-vendor-* jobs [18:16:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:01] marxarelli: hi! Any chance we can up the disk space quota for /var/lib/mysql from 256MB on the integration hosts? [18:28:26] they keep getting filled up and breaking all of fundraising's CiviCRM tests [18:28:29] ejegg: for which job? [18:28:42] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/7411/console [18:28:51] wikimedia-fundraising-civicrm [18:29:13] it's running on integration-slave-jessie-10XX [18:29:32] and I've been playing whack-a-mole with 240MB+ ibdata1 files [18:29:53] but they seem to expand again pretty quickly [18:30:05] so maybe a job to clear those out daily would be good too [18:30:45] https://phabricator.wikimedia.org/T205950 [18:30:49] ah, i see [18:30:51] it's on tmpfs [18:32:05] ejegg: so I noticed that /var/lib/mysql filled up again. From what I read yesterday that might be a problem with application code not closing transactions. It might be worth-while to do some exploration of the ibdata1 files and see how they filled so quickly. [18:32:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Ejegg) @thcipriani thanks! It seems to help temporarily, but those files grow back to consume most of the disk pretty quickly. Any chance we can increase the qu... [18:32:49] thcipriani: ah, interesting [18:33:09] sorry, submitted that phab comment before I saw ^^^ [18:33:13] https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/ was what i read yesterday [18:33:15] no worries :) [18:33:38] it could be just metadata from the tables, in which case we should probably just clear them out [18:33:55] unclosed txns would also cause problems with replication, right? [18:34:07] but if it'd be interesting if it was a bunch of rollback state [18:34:12] afaik replication is working well in production these days [18:34:24] dunno, honestly [18:34:55] maybe something test-specific is leading to all those hanging txns? [18:36:03] depends on the type of replication i think [18:36:15] i'm not sure whether we do row based or statement based [18:36:19] the latter would cause problems [18:37:20] marxarelli: we just switched over from statement based to row based, so we could ignore a few tables Civi uses to cache things [18:39:08] hmm, would be nice to have that custom version of innochecksum on an affected machine [18:43:04] (03CR) 10Dduvall: [C: 031] "Deployed to 12 mediawiki-quibble-vendor-* jobs and they seem to be functioning correctly. I'll go ahead and deploy to all 313 affected job" [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [18:43:22] oh hey, that was an easy compile :) [18:43:33] thcipriani: I guess it's mostly indexes [18:43:41] 13528 FIL_PAGE_INDEX [18:43:50] 398 FIL_PAGE_UNDO_LOG [18:44:34] heh, I just did the same thing [18:44:35] 445 FIL_PAGE_TYPE_ALLOCATED [18:45:05] (heck of a bug report!) [18:45:11] !log deploying I0fcd95427a6468213feb0be2ab7ab52f0d719f90 for all 313 affected jobs [18:45:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:46:39] ah, and none of the undos are active [18:46:39] undo state: 0 active, 250 cached, 0 to_free, 101 to_purge, 0 prepared, 47 other [18:47:23] so how does this keep filling up so quickly? also: I wonder how many tests use this db? [18:49:16] There's at least one chunky datasets we populate the dbs with in our setup scripts - all the zip codes in US+CAN, with lat+long+city+state [18:49:52] I think maybe one test depends on that though, so if we could use an alternate datafile for test setup that might help [18:51:13] I wonder if we could put this test in a container and run it on the docker nodes rather than on a host that we have to clean [18:51:52] thcipriani: that might be nice! there's a fair bit of setup scripted [18:52:05] but hopefully it wouldn't be too hard to containerize [18:52:57] If I were to start by just swapping out the dataset, what's the best way to detect that we're running on a test box? [18:53:17] This is code that would be hit before PHPUnit runs [18:53:45] ah, nvm, I think I see some defines in the test setup script that should serve [18:56:59] (03CR) 10Dduvall: [C: 032] Wipe entire workspace after builds of Docker based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [18:57:02] 10Release-Engineering-Team (Kanban), 10Surveys: Survey for Beta Cluster use cases - https://phabricator.wikimedia.org/T194818 (10greg) 05Open>03Resolved [18:57:27] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10Multi-Content-Revisions, 10Core Platform Team ( Code Health (TEC13)), and 3 others: Some PHPUnit tests consistently fail when collecting coverage report - https://phabricator.wikimedia.org/T183777 (10CCicalese_WMF) [18:58:29] 10Gerrit, 10Release-Engineering-Team: Upgrade gerrit to 2.15.4 - https://phabricator.wikimedia.org/T205784 (10Paladox) [18:59:25] (03Merged) 10jenkins-bot: Wipe entire workspace after builds of Docker based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [18:59:57] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.7-1 - https://phabricator.wikimedia.org/T204383 (10mmodell) [19:00:36] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Browser-Tests, 10Patch-For-Review, and 2 others: Browser Test Failures with Password Blacklist patches - https://phabricator.wikimedia.org/T204569 (10greg) a:03hashar >> How do we actually test this then? > I think @hashar needs to make a new... [19:01:39] 10Beta-Cluster-Infrastructure: Investigate setting up HTTPS directly on beta appservers - https://phabricator.wikimedia.org/T206158 (10Krenair) [19:02:08] 10Beta-Cluster-Infrastructure: Investigate setting up HTTPS directly on beta appservers - https://phabricator.wikimedia.org/T206158 (10Krenair) [19:02:12] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Technical-Debt, 10Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220 (10Krenair) [19:04:16] 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic: FY2017/18 Program 3 Outcome 1 Objective 2: Identify and find stewards for high-priority/high use code segment orphans - https://phabricator.wikimedia.org/T174091 (10greg) 05Open>03Resolved [19:04:18] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 1: The amount of orphaned code that is running Wikimedia โ€œproductionโ€ services is reduced. - https://phabricator.wikimedia.org/T174088 (10greg) [19:04:50] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 1: The amount of orphaned code that is running Wikimedia โ€œproductionโ€ services is reduced. - https://phabricator.wikimedia.org/T174088 (10greg) [19:04:52] 10Release-Engineering-Team, 10Epic, 10User-greg: FY2017/18 Program 3 Outcome 1 Objective 3: Define and steward a light-weight process for adopting or orphaning/sunsetting products and infrastructure. - https://phabricator.wikimedia.org/T174092 (10greg) 05Open>03Resolved a:03greg "lightweight" but https... [19:05:06] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3: Addressing technical debt - https://phabricator.wikimedia.org/T174087 (10greg) [19:05:49] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 2: Organizational technical debt is reduced. - https://phabricator.wikimedia.org/T174089 (10greg) [19:05:51] 10Release-Engineering-Team, 10Epic: FY2017/18 Program 3 Outcome 2 Objective 2: Define and implement a process to regularly address technical debt across the Foundation - https://phabricator.wikimedia.org/T174095 (10greg) 05Open>03Resolved JR has this and is working with eg Core Platform in the current fiscal. [19:07:39] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835 (10Paladox) 05Open>03Resolved The footer works both in GWTUI and polygerrit now! (showing the privacy notice and CoC) [19:08:22] 10Beta-Cluster-Infrastructure: Investigate setting up HTTPS directly on beta appservers - https://phabricator.wikimedia.org/T206158 (10Krenair) [19:12:01] andre__ hi just wanted to give you the fyi that upstream have added some new dialogs for new users to help them. Being https://gerrit-review.googlesource.com/c/gerrit/+/198270/1 and https://gerrit-review.googlesource.com/c/gerrit/+/198271/ [19:22:51] paladox: uh, nice! Thanks. Curious to see that in action [19:23:19] andre__ i took this https://phabricator.wikimedia.org/F26294871 [19:23:23] on the change screen [19:24:13] Hmm. Is that always correct in our cases? :-/ [19:24:22] I don't know enough of Gerrit to judge, I guess [19:24:35] that should be [19:24:39] thats how i use it [19:24:42] with plain git [19:25:35] andre__ judging by the other commit, if a user has no changes (ie a new user) then it will show them a dialog on what to do to get started [19:26:08] ah [19:26:19] ie https://gerrit-review.googlesource.com/c/gerrit/+/198270/1/polygerrit-ui/app/elements/change-list/gr-create-destination-dialog/gr-create-destination-dialog.html [19:26:27] andre__ so basically it will use the inline editor [19:26:30] to help users [19:26:31] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) 05Open>03Resolved Yup! I can see events here > https://grafana-la... [19:29:11] hmm nope, it will just show the user [19:29:15] i guess how to do it [19:29:17] * paladox trys it out [19:30:05] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10dduvall) p:05Triage>03High [19:30:20] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10dduvall) a:05dduvall>03None [19:30:58] 10Release-Engineering-Team (Kanban), 10Wikidata-Campsite-Iteration-โˆž, 10Release, 10Train Deployments, 10User-Addshore: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10Addshore) a:03Addshore [19:37:24] * addshore yells at CI for somehow not catching these errors.... [19:42:23] 10Release-Engineering-Team (Kanban), 10Wikidata-Campsite-Iteration-โˆž, 10Release, 10Train Deployments, 10User-Addshore: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10greg) p:05High>03Unbreak! [19:47:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Ejegg) Appears to be mostly indexes. Here's the output of the modified innochecksum util from here: https://bugs.mysql.com/bug.php?id=57611&files=1 0 bad check... [19:47:51] andre__ it looks like: https://phabricator.wikimedia.org/F26295418 and https://phabricator.wikimedia.org/F26295421 [19:49:32] 10Continuous-Integration-Config, 10Google-Code-in-2018: Add yourself to the Jenkins whitelist in Gerrit to trigger testing unit test failures or code style issues yourself - https://phabricator.wikimedia.org/T200778 (10Aklapper) [19:51:32] marxarelli: fyi it looks like only wikidatawiki needs to stay behind [19:51:39] 10Release-Engineering-Team (Kanban), 10Wikidata-Campsite-Iteration-โˆž, 10Release, 10Train Deployments, 10User-Addshore: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10Addshore) WikibaseQualityConstraints makes its own WikiPageEntityMetaDataLookup i... [19:51:43] WikibaseQualityConstraints is only on wikidatawiki and testwikidatawiki [19:52:05] * addshore quickly looks at any other entries in logstash [19:54:47] 10Release-Engineering-Team (Kanban), 10Wikidata-Campsite-Iteration-โˆž, 10Release, 10Train Deployments, and 2 others: Spike in fatals during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10Krinkle) [19:55:34] 10Release-Engineering-Team (Kanban), 10Wikidata-Campsite-Iteration-โˆž, 10Release, 10Train Deployments, and 2 others: Spike in fatals from WikiPageEntityMetaDataLookup during group1 deployment of 1.32.0-wmf.24 - https://phabricator.wikimedia.org/T206161 (10Krinkle) [19:57:06] greg-g: "Create a sub task" is so helpful in Phabricator :) [19:58:16] (03PS5) 10Legoktm: Update message to talk about "top level" instead of "file comment" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [19:58:24] (03CR) 10Legoktm: [C: 032] "..." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [19:59:08] I thought maybe if I append /form/10/ it'll turn into a normal task, but doesn't seem to be possible. [19:59:28] (03Merged) 10jenkins-bot: Update message to talk about "top level" instead of "file comment" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [20:00:06] (03CR) 10jenkins-bot: Update message to talk about "top level" instead of "file comment" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [20:01:25] addshore: k. thoughts on an eta? i can either keep wikidatawiki back and include it with group2 tomorrow, or wait on a fix. i'd prefer to wait for a fix unless it's going to push us past the window [20:01:44] how long is the window [20:01:51] i should be able to make a patch in the next few mins tbh [20:02:05] another hour [20:02:11] right on! [20:02:17] i'll wait then :) [20:03:05] greg-g: er, looks like we have overlapping windows today. was that intentional? [20:04:14] ah, no matter ( no parsoid deploy today.) [20:07:34] marxarelli: they're like that on wednesdays :/ [20:07:45] marxarelli: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseQualityConstraints/+/464407/ should be it [20:07:48] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Jdforrester-WMF) >>! In T206046#4637493, @Krinkle wrote: > Deja vu โ€“ย {T127504}. Yeah, as soon as Lego mentioned it I facepalmed... [20:08:00] greg-g: ah, ok :) [20:08:07] addshore: great! [20:08:28] twentyafterfour: is there a way to fix this "people make subtasks of the train task then become "release tasks" that can't be changed" problem? [20:09:10] greg-g: oh, whoops. i didn't know about that :/ [20:09:18] marxarelli: do we have a way to stage .23 and that patch on mwdebug? [20:09:25] if so I could verify it as well before rolling forward [20:09:29] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Legoktm) >>! In T206046#4639882, @Jdforrester-WMF wrote: >>>! In T206046#4637493, @Krinkle wrote: >> Deja vu โ€“ย {T127504}. > > Y... [20:10:03] addshore: hmm, yes but i don't recall how to do mwdebug deploys [20:10:05] thcipriani: ^ ? [20:10:26] (it's been a while) [20:10:48] legoktm: Sure, but it wasn't *your* mistake each time. :-) [20:10:49] ssh mwdebug100X; scap pull [20:11:09] (In this case, mwdebug2001 please.) [20:11:21] James_F: :p I think it should be easier to fix this time since we're already moving to PHP 7 evyerwhere else [20:11:36] * James_F nods. [20:11:54] marxarelli: stage it on deploy host, pull from mwdebug [20:12:16] oh, but with the old version... [20:12:17] right oh [20:12:24] i guess? [20:12:29] then we might as well stage it for wmf.24 i suppose? [20:13:40] it's not going to give us as much certainty in the fix in the context of wmf.24 if we deploy for wmf.23... is my thinking [20:14:20] * addshore is confused, the fix is only relevant for .24 [20:14:30] yep yep [20:14:41] is there no was to bump the version of mw bring run for wikidata wiki just on the debug host (and have the patch)? :( [20:16:12] how is that code path not touched in CI [20:16:17] * addshore facepalms [20:17:22] yes i should just be able to bump wikiversions for wikidatawiki locally, apply the patch locally, and then scap pull on mwdebug100{1,2} [20:18:38] if it's all good, i'll revert the local changes and we'll do things properly through gerrit [20:19:28] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Krinkle) Hm.. would it make sense to "quickly" switch to HHVM first? It's not something I think we should do in general, but gi... [20:20:36] * addshore is fine with whatever :) [20:20:48] marxarelli: give me a ping if / when it is testable, / pre deploy :) [20:21:15] will do! [20:23:46] addshore: k. patch and wikiversions.json change for wikidatawiki is on mwdebug100x [20:24:19] legoktm: btw ty! [20:24:51] marxarelli: 100x? which is x? :P [20:25:05] mwdebug1001 and mwdebug1002 :) [20:25:18] oooh, both! [20:25:31] @greg-g: there isn't an easy way to fix that, no [20:25:39] maybe a herald rule can do it... let me see [20:28:50] marxarelli: the test url i had works [20:29:01] but on special:version on both servers it still shows .23 :/ [20:29:06] hmm [20:29:13] maybe it's hitting mwdebug2001 [20:29:52] 2001 also shows .23 [20:29:54] for me [20:29:55] i'll update on those two servers as well - mwdebug200x for x in {1..2} :) [20:32:57] addshore: ah, wikiversion.php was not changed [20:33:10] sec [20:33:13] :) [20:35:08] addshore: k. it should be right on all 4 mwdebug servers [20:35:29] (now) :) [20:35:34] marxarelli: both special version and my test url look good :) [20:35:37] lets do this! [20:35:48] great! [20:36:27] greg-g: maybe https://phabricator.wikimedia.org/H301 will do the trick [20:36:36] addshore: oh i just noticed your patch is for master. can you submit one for the 1.32.0-wmf.24 branch please? [20:36:44] there is one for .24 too [20:36:48] oh. :) [20:36:50] same change-id [20:37:15] great! i'll +2 [20:38:07] twentyafterfour: interesting, so 'type' is mutable via Herald? cool. [20:38:31] Krinkle: yes and also via conduit or batch editor [20:39:03] 10Release-Engineering-Team (Kanban), 10Release: Testing a release subtask - https://phabricator.wikimedia.org/T206167 (10mmodell) [20:39:13] https://phabricator.wikimedia.org/T206167 herald did the trick [20:39:26] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Jdforrester-WMF) That'd be fine, though moving to 7.x directly is slightly more preferable than moving to HHVM and thence to 7.x... [20:39:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T191072 (10mmodell) [20:39:44] 10Release-Engineering-Team (Kanban), 10Release: Testing a release subtask - https://phabricator.wikimedia.org/T206167 (10mmodell) 05Open>03Invalid [20:39:56] 10Release-Engineering-Team (Kanban), 10Release: Testing a release subtask - https://phabricator.wikimedia.org/T206167 (10mmodell) Seems to work. [20:40:45] twentyafterfour: the system we use for creating the weekly blocking task, is it capable of self-referencing? If so, it could also link to something like https://phabricator.wikimedia.org/maniphest/task/edit/form/46/?parent=191072 [20:40:52] in the task description or some such [20:41:35] yeah that could work [20:41:53] The text in the "details" section for deployment tasks is actually dynamic not statically generated [20:42:04] it's a template rendered at runtime [20:42:16] Oh wow. Interesting. [20:42:24] I assumed it was a bot substituting it via the API or some such [20:42:30] PHP? [20:43:22] Krinkle: php, yes: https://phabricator.wikimedia.org/source/phab-extensions/browse/wmf%252Fstable/src/customfields/ReleaseDetailsCustomField.php [20:44:07] it's a virtual field that gets added to that type of task which has custom implementation [20:44:36] so it grabs two small variables from the release version and release date fields to generate all the rest of the content [20:44:56] just a timestamp and a version string which get parsed into all that stuff on the release details [20:46:02] there is a bit of python to generate a series of the tasks via conduit but most of the stuff is in php/phabricator extension rather than statically generated [20:46:24] twentyafterfour: thanks! [20:56:24] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:16] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [21:11:42] twentyafterfour: nice [21:54:00] Only took 3 years :D https://github.com/pear/Net_SMTP/issues/24 [21:56:19] no_justification hi :) [22:04:11] no_justification: Fun. [22:04:53] 35 months, sheesh :) [22:06:24] And it only took me 25 days to notice the first comment [22:06:29] (I could lie and say 9, but I didn [22:06:34] 't notice til now) [22:07:11] Filed as https://phabricator.wikimedia.org/T206176 [22:10:08] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Packaging, and 2 others: Update Debian Package for Scap to 3.8.7-1 - https://phabricator.wikimedia.org/T204383 (10mmodell) [23:07:53] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) a:03jcrespo [23:12:27] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:16] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [23:19:04] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Add Code Health Metrics Working Group members to Code Health mailing list - https://phabricator.wikimedia.org/T205132 (10Jrbranaa) Forgot my password :-( Waiting for reset. [23:21:23] 10Release-Engineering-Team (Kanban): Add Code Stewardship review to Review Queue process - https://phabricator.wikimedia.org/T203698 (10Jrbranaa)