[01:51:18] Krinkle: npm yes. npm6 is ready to go, I just haven't had time to babysit the rollout [01:52:06] k [02:49:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T191072 (10Krinkle) [02:52:23] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T191072 (10Krinkle) [03:16:12] 10Continuous-Integration-Infrastructure, 10LuaSandbox: Run LuaSandbox tests against PHP with ZTS - https://phabricator.wikimedia.org/T206591 (10Legoktm) p:05Triage>03Lowest [03:26:53] 10Continuous-Integration-Infrastructure, 10LuaSandbox, 10wikidiff2: Test against PHP extensions in 32-bit - https://phabricator.wikimedia.org/T188482 (10Legoktm) [06:17:13] Project beta-scap-eqiad build #224418: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224418/ [06:32:37] Yippee, build fixed! [06:32:37] Project beta-scap-eqiad build #224419: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224419/ [08:06:09] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet error on deployment-mwmaint01 - https://phabricator.wikimedia.org/T206598 (10Krenair) [08:15:17] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet error on deployment-mwmaint01 - https://phabricator.wikimedia.org/T206598 (10Krenair) [10:09:49] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:15:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:17:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:36:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:38:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:40:53] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:50:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [10:50:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [10:52:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [10:53:24] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [11:06:30] 10Project-Admins, 10User-Urbanecm: Do not add #user-urbanecm to task Urbanecm created - https://phabricator.wikimedia.org/T206618 (10Urbanecm) [11:09:15] 10Project-Admins: Do not add #user-urbanecm to task Urbanecm created - https://phabricator.wikimedia.org/T206618 (10Urbanecm) [11:33:56] 10Deployments: Check 'Check endpoints for mwdebug2002.codfw.wmnet' failed: /wiki/{title} (Main Page) is WARNING: Test Main Page responds with unexpected body - https://phabricator.wikimedia.org/T206620 (10zeljkofilipin) [11:35:54] (03PS1) 10Reedy: Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) [11:38:39] (03CR) 10jerkins-bot: [V: 04-1] Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [11:40:53] (03PS2) 10Reedy: Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) [11:43:52] (03CR) 10jerkins-bot: [V: 04-1] Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [11:44:30] (03PS3) 10Reedy: Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) [11:47:48] (03CR) 10jerkins-bot: [V: 04-1] Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [11:48:14] (03CR) 10Reedy: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [11:52:44] 10Release-Engineering-Team (Kanban), 10Scap: Check 'Check endpoints for mwdebug2002.codfw.wmnet' failed: /wiki/{title} (Main Page) is WARNING: Test Main Page responds with unexpected body - https://phabricator.wikimedia.org/T206620 (10zeljkofilipin) [12:00:35] 10Phabricator, 10Project-Admins: Do not add #user-urbanecm to task Urbanecm created - https://phabricator.wikimedia.org/T206618 (10MGChecker) [12:03:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) [12:03:20] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) p:05Triage>03High [12:05:26] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) [12:07:19] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10User-zeljkofilipin: Develop set of metrics to assess incident reports/post mortems - https://phabricator.wikimedia.org/T206622 (10zeljkofilipin) [12:07:29] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10User-zeljkofilipin: Develop set of metrics to assess incident reports/post mortems - https://phabricator.wikimedia.org/T206622 (10zeljkofilipin) p:05Triage>03Normal [12:10:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [12:10:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [12:12:08] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q2 Selenium framework improvements - https://phabricator.wikimedia.org/T206624 (10zeljkofilipin) [12:12:50] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986 (10zeljkofilipin) [12:12:53] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q2 Selenium framework improvements - https://phabricator.wikimedia.org/T206624 (10zeljkofilipin) [12:13:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q2 Selenium framework improvements - https://phabricator.wikimedia.org/T206624 (10zeljkofilipin) p:05Triage>03Normal [12:13:48] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986 (10zeljkofilipin) [12:22:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q2 Selenium framework improvements - https://phabricator.wikimedia.org/T206624 (10zeljkofilipin) [12:25:28] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986 (10zeljkofilipin) [12:25:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) 05Open>03Resolved Moved unresolved tasks to {T206624}. 😦 [12:25:46] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q2 Selenium framework improvements - https://phabricator.wikimedia.org/T206624 (10zeljkofilipin) [12:33:04] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Deployments of MediaWiki with scap cause a spam of "web request took longer than 60 seconds and timed out" - https://phabricator.wikimedia.org/T204871 (10zeljkofilipin) [12:33:34] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Deployments of MediaWiki with scap cause a spam of "web request took longer than 60 seconds and timed out" - https://phabricator.wikimedia.org/T204871 (10zeljkofilipin) Logs are full of this error message. [13:04:42] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [13:54:32] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [13:56:35] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [13:57:20] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [13:58:00] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [13:58:55] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [14:02:04] reminder, no puppet merges for the next hour while data center switchover is taking place [14:06:22] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [14:07:16] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [14:08:05] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [14:13:24] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10zeljkofilipin) [14:43:17] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10Marostegui) This was done successfully and new wikis are now live on eqiad. What is pending now is: - Run the DNS changes for wiki... [14:45:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:47:21] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:50:28] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:53:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:01:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10greg) [15:26:26] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:12] RECOVERY - DPKG on contint2001 is OK: All packages OK [15:50:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10greg) [16:01:12] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [16:09:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Quibble: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 (10dduvall) p:05Triage>03High a:03dduvall [16:12:30] (03PS1) 10Thiemo Kreuz (WMDE): Edit Project Config [extensions/FileImporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/465653 [16:12:45] (03Abandoned) 10Thiemo Kreuz (WMDE): Edit Project Config [extensions/FileImporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/465653 (owner: 10Thiemo Kreuz (WMDE)) [16:22:10] Project beta-scap-eqiad build #224457: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224457/ [16:22:18] Project beta-update-databases-eqiad build #28943: 04FAILURE in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28943/ [16:23:06] Project beta-scap-eqiad build #224458: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224458/ [16:24:37] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog: selenium-daily-beta-Popups CI job failing - https://phabricator.wikimedia.org/T206640 (10Jdlrobson) [16:26:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog (Tracking): selenium-daily-beta-Popups CI job failing - https://phabricator.wikimedia.org/T206640 (10Jdlrobson) These appear to be passing again but seems like a problem in the infrastructure. [16:36:46] Project beta-scap-eqiad build #224459: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224459/ [16:46:15] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [16:50:29] Project beta-scap-eqiad build #224460: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224460/ [16:52:22] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:15] Project beta-scap-eqiad build #224461: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224461/ [17:06:34] cp: error writing '/tmp/scap_l10n_149532886/l10n_cache-zh-cn.cdb': No space left on device [17:11:17] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10LarsWirzenius) [17:13:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Onboarding liw - https://phabricator.wikimedia.org/T206466 (10greg) [17:15:41] Project beta-scap-eqiad build #224462: 15ABORTED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224462/ [17:15:50] !log cowboy-coding on deployment-deploy01 to solve scap fatal error dcheck failing to catch fatals [17:15:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:16:06] I aborted beta-scap-eqiad [17:17:45] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10greg) >>! In T191183#4653436, @thcipriani wrote: > This is probably something we should enforce somehow (jenkins? some tool to be created to upload?) before exposing this feature b... [17:22:13] Yippee, build fixed! [17:22:13] Project beta-update-databases-eqiad build #28944: 09FIXED in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28944/ [17:27:50] (03PS1) 10Dduvall: Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) [17:28:50] (03CR) 10jerkins-bot: [V: 04-1] Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [17:32:36] 10Phabricator: Do not add #user-urbanecm to task Urbanecm created - https://phabricator.wikimedia.org/T206618 (10MGChecker) 05Open>03Resolved a:03MGChecker [17:33:12] (03PS4) 10Dduvall: Revert "Use a volume under workspace for /tmp in docker containers" [integration/config] - 10https://gerrit.wikimedia.org/r/463852 [17:35:05] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [17:35:21] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [17:36:44] (03PS2) 10Dduvall: Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) [17:37:47] (03CR) 10jerkins-bot: [V: 04-1] Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [17:50:09] (03PS1) 10Dduvall: Ensure Docker containers are properly killed following builds [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) [17:51:31] (03CR) 10jerkins-bot: [V: 04-1] Ensure Docker containers are properly killed following builds [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [18:11:42] thcipriani: just a heads up in case you missed it https://phabricator.wikimedia.org/T205563#4655012 [18:18:04] chasemp: I saw, thanks for the poke, I'll fiddle with this today [18:21:47] 10Continuous-Integration-Infrastructure, 10translatewiki.net: L10n-bot should not force-merge / override Jenkins (breaks the build) - https://phabricator.wikimedia.org/T91707 (10chasemp) a:05chasemp>03None Thanks, I misunderstood the scope here. Leaving open :) [18:38:08] Yippee, build fixed! [18:38:08] Project beta-scap-eqiad build #224463: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224463/ [18:38:59] (03PS3) 10Dduvall: Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) [18:39:31] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10mmodell) Update: @thcipriani and I just did a pairing session to get to the bottom of this. Our conclusion: something in scap or the environment is eating stderr.... [18:39:33] twentyafterfour: what was the outcome of the cowboy coding? ) [18:39:54] ah, I see, that wasn't in response to the diskfull error.... [18:40:48] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10mmodell) >>! In T121597#4652873, @Krinkle wrote: > Looks like it might not be working as intended. > > Here's what I did: > > * Disabled `beta-scap-eqiad`. > *... [18:44:34] greg-g: the disk full error is a knock-on effect of https://phabricator.wikimedia.org/T205463 scap fails a few times, leaves behind a bunch of tmp files (which we should fix), disk fills. Immediate-term solution is to do: sudo rm -rf /tmp/scap_l10n* [18:45:01] short-term solution is to put the removal of those tmp files in a "finally" block [18:45:28] ack, thanks [18:46:40] (03PS2) 10Dduvall: Ensure Docker containers are properly killed following builds [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) [18:47:58] well. shoot. I guess it can't remove those files since it needs some kind of sudo access to do so :P [18:48:03] * thcipriani thinks of other solution [18:54:53] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10Krinkle) @mmodell Both are actually loaded in Beta Cluster, and should be. The `-labs` file is a light layer of overrides atop the main file. See also [file comme... [19:04:02] 10Deployments, 10I18n: Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10greg) p:05Triage>03Normal [19:08:03] 10Deployments, 10Language-Team, 10I18n: Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10greg) [19:19:43] twentyafterfour hi, would using https://phabricator.wikimedia.org/conduit/method/user.ldapquery/ list all users in ldap? [19:20:54] 10Deployments, 10Language-Team, 10I18n: Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10Nemo_bis) If scap now has problems with the l10n updates, it should be reverted to the version which did not have such problems, so that l10n upda... [19:23:12] 10Deployments, 10Language-Team, 10I18n: Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10siebrand) [19:23:52] 10Deployments, 10Language-Team, 10I18n: Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10greg) >>! In T206694#4656195, @Nemo_bis wrote: > If scap now has problems with the l10n updates, it should be reverted to the version which did no... [19:26:03] twentyafterfour: not terribly important, but https://phabricator.wikimedia.org/T206655 (the first of the new train blocker series) is linking to the wrong previous task [19:26:25] the rest are correct, afaict (from the my few clicks ;) ) [19:38:21] !log restarted ORES celery workers on ores2003 (~17:00), ores200* (17:05) [19:38:26] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:37] [oops, ->operations!] [19:38:39] awight, you might want to log that one in ops [19:38:42] :D [19:38:44] too late [19:38:48] o/ it's been a while, happy to say [19:38:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:39:05] Oh well now it'll remind us in our beta SAL too :D [19:45:59] heh, it worked here (cc mutante ) but took a while. I wonder if stashbot was just overloaded for a bit [20:01:41] 10Phabricator, 10Operations: raid_handler.py using deprecated conduit api - https://phabricator.wikimedia.org/T206697 (10Paladox) [20:03:13] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [20:03:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-OAuth, 10MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Wikimedia-production-error: [OAuth] PHP Fatal Error: Call to undefined method MediaWiki\Extensions\OAuth\MWOAuthDAOAccessControl::getForHtm... - https://phabricator.wikimedia.org/T204757 [20:03:26] 10Phabricator, 10Operations, 10Operations-Software-Development, 10Technical-Debt: Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api - https://phabricator.wikimedia.org/T159045 (10Paladox) [20:03:30] 10Phabricator, 10Operations: raid_handler.py using deprecated conduit api - https://phabricator.wikimedia.org/T206697 (10Paladox) [20:03:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-OAuth, 10MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Wikimedia-production-error: [OAuth] PHP Fatal Error: Call to undefined method MediaWiki\Extensions\OAuth\MWOAuthDAOAccessControl::getForHtm... - https://phabricator.wikimedia.org/T204757 [20:03:53] 10Phabricator, 10Operations, 10Operations-Software-Development, 10Technical-Debt: Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api - https://phabricator.wikimedia.org/T159045 (10Paladox) Need to also migrate from "project.query" to "project.search" [20:32:18] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10Paladox) We won't be able to use git lfs on cobalt as it is using jessie whereas the git-lfs package is in stretch-backports+. We could enforce it so users can only upload there i... [20:43:58] greg-g: I think your script has a bug/easter egg [20:43:59] Release Date 1555286400 [20:55:17] where? [20:55:19] Reedy: ^ [21:17:21] greg-g: https://phabricator.wikimedia.org/T206680 and every one before it [21:17:25] Release Version 1.33.0-wmf.25 Release Date 1554681600 [21:18:40] Dunno if phab has mangled something [21:20:00] the dates in the description are correct... cc twentyafterfour ^ [21:23:09] Project beta-scap-eqiad build #224474: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224474/ [21:23:18] Project beta-update-databases-eqiad build #28948: 04FAILURE in 3 min 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28948/ [21:23:54] 10Phabricator-Sprint-Extension: Wikimedia Phabricator sprint extension uses removed ManiphestEditStatusCapability - https://phabricator.wikimedia.org/T206538 (10Aklapper) Thanks for taking a look at the code! If you fancy fixing this, you are very welcome to use [[ https://www.mediawiki.org/wiki/Developer_access... [21:24:55] 10Phabricator-Sprint-Extension: Wikimedia Phabricator sprint extension uses removed ManiphestEditStatusCapability - https://phabricator.wikimedia.org/T206538 (10Aklapper) p:05Triage>03Lowest [21:25:02] 10Phabricator-Sprint-Extension: Unhandled Exception ("PhutilMissingSymbolException") - https://phabricator.wikimedia.org/T206489 (10Aklapper) p:05Triage>03Lowest [21:33:28] 10Phabricator, 10Operations, 10Operations-Software-Development, 10Technical-Debt: Update Puppet repo code that uses deprecated maniphest.update/.createtask/.query Conduit API - https://phabricator.wikimedia.org/T159045 (10Aklapper) [21:38:35] Yippee, build fixed! [21:38:35] Project beta-scap-eqiad build #224475: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224475/ [22:18:49] I don't see what's wrong with those deployment blocker tasks? [22:19:13] is it just the formatting of the release date field? [22:19:33] twentyafterfour: it's showing in unix time [22:19:40] Which isn't human readable [22:19:56] Hasn't it been like that? [22:20:14] Can't say I've ever noticed [22:20:19] hmm [22:20:33] I wouldn't worry then :) [22:20:35] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10greg) >>! In T191183#4656413, @Paladox wrote: > We won't be able to use git lfs on cobalt as it is using jessie whereas the git-lfs package is in stretch-backports+. That's simply... [22:20:54] 1.32.0-wmf.9 is the same [22:20:55] https://phabricator.wikimedia.org/T191055 [22:20:56] indeed it shouldn't be visible really [22:21:28] Why does it jump from .9 to 1.33? [22:22:27] Yippee, build fixed! [22:22:27] Project beta-update-databases-eqiad build #28949: 09FIXED in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28949/ [22:23:41] Reedy: that part is a bug [22:23:47] heh [22:23:48] which I don't quite understand [22:23:53] computers suck [22:26:12] ^ /me signs affadavit to affirm [22:39:57] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:42:44] https://integration.wikimedia.org/zuul/ has a lot of jobs [22:42:46] Not a lot of action [22:46:11] thcipriani: ^^ [22:51:30] hrm, nothing abnormal in zuul logs, really [22:51:47] picking up again [22:51:51] James blames Timo [22:52:14] No no, I said it was operating "normally". [22:52:18] :) [22:52:19] No blame. :-) [22:52:50] I don't understand why that page is so big. [22:53:07] Looks like it's creating a more complex tree than needed. [22:53:38] It creates the trees A, AB, ABC, ABCD, … ABCDEFGHIJ. That's what's intended, surely? [22:54:24] Yeah, I see now. It's inferring the relations but unlike the submit pipeline, for test the three isn't blocking. [22:54:30] it's actually running all the entries fine. [22:54:55] It's just the known issues with zuul/gearman scheduling where we quite often have over 50% idle capacity, but nothing is scheduled there [22:55:02] * James_F nods. [22:56:53] 10Project-Admins: Create component project for GraphQL - https://phabricator.wikimedia.org/T206499 (10Aklapper) Thanks, because that means it should be created under https://phabricator.wikimedia.org/project/subprojects/525/ instead of https://phabricator.wikimedia.org/project/subprojects/703/ I guess [23:15:42] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10Tgr) UX-wise a single central place for profile images is obviously preferable, so using Phabricator makes a lot of sense. (Having some way to store a profile image in your Wikimed... [23:17:04] (03CR) 10Thcipriani: [C: 031] "lgtm, reason for change makes sense." [integration/config] - 10https://gerrit.wikimedia.org/r/463852 (owner: 10Dduvall) [23:19:24] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:10] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, and 2 others: [Epic] ORES should use a git large file plugin for storing serialized binaries - https://phabricator.wikimedia.org/T171619 (10awight) [23:25:18] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, and 2 others: [Epic] ORES should use a git large file plugin for storing serialized binaries - https://phabricator.wikimedia.org/T171619 (10awight) [23:44:08] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1