[00:01:09] 10Continuous-Integration-Config, 10LibUp, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10phan, and 2 others: Upgrade phan to 0.10.1 and remove phan-taint-check-plugin - https://phabricator.wikimedia.org/T248630 (10Daimona) Meh, it's taint-check messing up with phan. I'll take a look tomorrowi... [00:48:52] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:45:49] 10LibUp: libup KeyError 'actions' in npm_audit_fix - https://phabricator.wikimedia.org/T228173 (10Legoktm) 05Resolved→03Open This is happening again. [02:09:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:08] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:13] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:19:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93835 bytes in 0.952 second response time [02:19:58] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 94297 bytes in 1.078 second response time [02:20:04] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52391 bytes in 0.978 second response time [06:18:11] 10Gerrit, 10DBA: Investigate Gerrit troubles to reach the MariaDB database - https://phabricator.wikimedia.org/T247591 (10Marostegui) @hashar any more issues with this? [09:55:41] 10Phabricator: Disable Herald rules H66, H236, H313 - https://phabricator.wikimedia.org/T248662 (10Aklapper) p:05Triage→03Low [10:00:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:08] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:15] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:40] 10Phabricator: Disable Herald rules H66, H236, H313 - https://phabricator.wikimedia.org/T248662 (10Aklapper) Meh. `sudo /srv/phab/phabricator/bin/policy unlock --edit` throws `EXCEPTION: (Exception) Object edit policy can not be unlocked because this object does not have a mutable edit policy.` `sudo /srv/pha... [10:02:43] 10Gerrit, 10DBA: Investigate Gerrit troubles to reach the MariaDB database - https://phabricator.wikimedia.org/T247591 (10hashar) 05Stalled→03Resolved That mysteriously stopped even before the m2-master database got upgraded and hasn't occurred since. I guess that rules out the database. Maybe there is wa... [10:02:45] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), and 2 others: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) [10:03:01] 10Phabricator: Disable Personal Herald rules H66, H236, H313 as the user currently cannot access Phab - https://phabricator.wikimedia.org/T248662 (10Aklapper) [10:04:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), and 2 others: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) 05Stalled→03Declined It is unlikely we... [10:08:06] (03CR) 10Daniel Kinzler: [C: 03+2] Update UnrecognizedAnnotation sniff for stable interface policy tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/580356 (https://phabricator.wikimedia.org/T247836) (owner: 10DannyS712) [10:08:43] (03Merged) 10jenkins-bot: Update UnrecognizedAnnotation sniff for stable interface policy tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/580356 (https://phabricator.wikimedia.org/T247836) (owner: 10DannyS712) [10:15:49] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93837 bytes in 1.053 second response time [10:15:58] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 94257 bytes in 1.037 second response time [10:16:03] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52423 bytes in 1.086 second response time [11:48:27] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2020): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Aklapper) [12:18:15] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [12:18:47] Krinkle: Reedy: Fancy applying an edit to wikitech's common.js for me? [12:21:01] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [12:28:17] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [12:31:02] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [12:47:13] hoo: I can do it [12:48:09] Cool [12:48:35] Can you change the definition of localTz in handleCalendarPage to localTz = startTime.getTimezoneOffset() / -60; [12:48:55] And move it down below the initialization of startTime [12:49:26] * Amir1 looks [12:49:39] Right now the script always show the current timezone, but for example EU has a new timezone from Monday on [12:49:47] so that should be shown, not the current one [13:08:24] 10Project-Admins: Create Project: Section-Name-Diff - https://phabricator.wikimedia.org/T246484 (10Aklapper) 05Open→03Resolved a:03Aklapper As subprojects have some disadvantages (membership as this is only "for a certain time"), I created a top-level project: Requested public project #section-name-diff h... [13:25:54] (03CR) 10Hashar: [C: 03+2] "Deployed. The job ran in just 8 seconds instead of 15-19 seconds. I guess all those print statements each require a network roundtrip." [integration/config] - 10https://gerrit.wikimedia.org/r/582856 (owner: 10Hashar) [13:26:50] (03Merged) 10jenkins-bot: jjb: reduce disk check job disk usage on master [integration/config] - 10https://gerrit.wikimedia.org/r/582856 (owner: 10Hashar) [13:33:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Core Platform Team, 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel) [13:34:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Core Platform Team, 10CPT Initiatives (API Integration Tests), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel) [13:53:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10Anomie) Untagging #core_platform_team as this is alre... [14:35:41] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Rileych: Thanks! I have added you. (Usual default disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#New_proj... [14:46:31] 10Project-Admins, 10User-DannyS712: Create Project: Named-References-VE - https://phabricator.wikimedia.org/T246485 (10Aklapper) @ifried: Could you please reply, before declining this due to no feedback? Thanks! :) [15:52:53] (03PS3) 10Lars Wirzenius: Fix scap sync without -w (--canary-wait-time) [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 [16:10:42] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10Krinkle) Keeping this open until the next release/deploy of Quibble. [16:10:48] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10Krinkle) a:05Krinkle→03None [16:11:33] 10Continuous-Integration-Config, 10Parsoid-PHP: Set up extension tests for Parsoid repo - https://phabricator.wikimedia.org/T227352 (10cscott) [16:16:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Jenkins: Migrate CI Jenkins artifacts to a non wikimedia.org domain - https://phabricator.wikimedia.org/T248696 (10hashar) [16:16:49] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T247775 (10thcipriani) [16:17:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Jenkins: Migrate CI Jenkins artifacts to a non wikimedia.org domain - https://phabricator.wikimedia.org/T248696 (10hashar) That is not ideal, requests for artifacts get redirected to the resource domain with a per user crafted URL. T... [16:17:30] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T247776 (10thcipriani) [16:19:05] Hi guys. I'm Karl Wettin (WMSE). I have a couple of questions regarding Blubber in CI. What's acceptable and not regarding insecure images, how you guys build the Docker images, where we pick up the services to be Blubberized, what we trust in third party repositories, and so on. Would it be possible to get a quick voice/video call with someout about this? [16:20:15] I'm in CET-zone, but available pretty much at your convenience. [16:23:56] kalle: short answer is: for things going to production, we only accept base images produced and hosted by us/the WMF. [16:31:21] greg-g: This is for the Wikispeech project. We use services that we developed ourselfs, but also services developed by third parties and available at GitHub. Building these images include cloning the repos and copying them to the Docker image. When does this become a problem? Should we for example perhaps host forks at trusted revisions? Mirror them in Gerrit? How much do we need to review them? Etc. [16:38:16] kalle: remind me, is this work already in production/in use in any way? [16:48:18] greg-g: Not in production. We're running services (Text-To-Speech etc) out of our manually configured VMs on wmflabs, available for those (read: us) that do local testing of the extension. [16:48:42] Blubberization is a part of us moving towards acceptable beta. [16:49:25] gotcha [16:56:03] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [17:08:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Grant "contint-roots" and "releasers-mediawiki" to user jforrester - https://phabricator.wikimedia.org/T248597 (10Volans) 05Open→03Resolved Confirmed all looks good so far, resolving. [19:06:22] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:16:12] (03PS1) 10Jforrester: deployments-calendar: Prefix {{ircnick}} lines with \n; ask for puppet patches in puppet SWAT [tools/release] - 10https://gerrit.wikimedia.org/r/584034 [19:24:01] (03CR) 10Jforrester: [C: 03+2] deployments-calendar: Prefix {{ircnick}} lines with \n; ask for puppet patches in puppet SWAT [tools/release] - 10https://gerrit.wikimedia.org/r/584034 (owner: 10Jforrester) [19:24:38] (03Merged) 10jenkins-bot: deployments-calendar: Prefix {{ircnick}} lines with \n; ask for puppet patches in puppet SWAT [tools/release] - 10https://gerrit.wikimedia.org/r/584034 (owner: 10Jforrester) [19:24:43] (03Abandoned) 10Jforrester: MakeWmfBranch: Support MW_VERSION as well as $wgVersion [tools/release] - 10https://gerrit.wikimedia.org/r/507112 (owner: 10Jforrester) [19:25:00] (03PS5) 10Jforrester: Drop merge-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/563498 [19:25:09] (03CR) 10Jforrester: [C: 03+2] Drop merge-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/563498 (owner: 10Jforrester) [19:26:10] (03Merged) 10jenkins-bot: Drop merge-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/563498 (owner: 10Jforrester) [19:26:30] (03PS4) 10Jforrester: Stop branching ParsoidBatchAPI [tools/release] - 10https://gerrit.wikimedia.org/r/563462 (https://phabricator.wikimedia.org/T242430) (owner: 10Reedy) [19:26:52] (03PS4) 10Jforrester: Add the very dangerous --delete argument to mass delete branches [tools/release] - 10https://gerrit.wikimedia.org/r/575571 (owner: 1020after4) [19:41:44] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:51:39] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Quibble, 10Patch-For-Review, 10Regression: Jenkins no longer fails build if mw-error.log is non-empty - https://phabricator.wikimedia.org/T246358 (10Pchelolo) [20:10:48] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE): Cannot push to gerrit - https://phabricator.wikimedia.org/T248710 (10Gbahdeyboh) [20:14:39] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE): Cannot push to gerrit - https://phabricator.wikimedia.org/T248710 (10Reedy) You look to have made some sort of change ontop of REL1_34, and are trying to merge the whole branch into master... [20:43:31] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Milimetric) Recent events have compelled me to adopt graphoid. I will personally maintain it until w... [21:09:45] (03CR) 1020after4: [C: 03+2] "Nice work." [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 (owner: 10Lars Wirzenius) [21:13:18] (03CR) 1020after4: [C: 03+1] Add detailed release procedure documentation [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [21:14:45] (03Merged) 10jenkins-bot: Fix scap sync without -w (--canary-wait-time) [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 (owner: 10Lars Wirzenius) [22:40:35] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE): Cannot push to gerrit - https://phabricator.wikimedia.org/T248710 (10Gbahdeyboh) @Reedy I created a new branch and committed my changes to that branch, what I am trying to push is the branch I newly created [22:49:15] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE): Cannot push to gerrit - https://phabricator.wikimedia.org/T248710 (10Reedy) What commands did you enter? [22:59:27] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Improve Basic Kubernetes Knowledge within RelEng - https://phabricator.wikimedia.org/T245653 (10DC_Slagel) a:05jeena→03mmodell [22:59:56] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Improve Basic Kubernetes Knowledge within RelEng - https://phabricator.wikimedia.org/T245653 (10DC_Slagel) a:05mmodell→03jeena [23:15:36] Reedy ^ [23:15:41] (i think thats spam?) [23:16:03] oh [23:16:07] Yeah.. Doesn't look good [23:16:08] Oh? [23:16:26] Yeh, nvm. I think they made a mistake, sorry for the spam myself :) [23:40:15] 10Project-Admins, 10User-DannyS712: Create Project: Named-References-VE - https://phabricator.wikimedia.org/T246485 (10ifried) I apologize for not responding sooner. I am fine with any suggested name, so long as it is a unique project tag that is not being used by any other teams. [23:41:35] 10Project-Admins: Create Project: Section-Name-Diff - https://phabricator.wikimedia.org/T246484 (10ifried) Thank you, @Aklapper! I will add this tag to the relevant tickets. [23:54:32] 10Project-Admins, 10User-DannyS712: Create Project: Named-References-VE - https://phabricator.wikimedia.org/T246485 (10DannyS712) >>! In T246485#6006552, @ifried wrote: > I apologize for not responding sooner. I am fine with any suggested name, so long as it is a unique project tag that is not being used by an...