[01:44:40] 10Gerrit, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Krinkle) [01:45:07] 10Gerrit, 10Release-Engineering-Team-TODO, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Krinkle) Seen a few times a day or every other day it seems. [01:51:37] 10Continuous-Integration-Infrastructure, 10Cleanup, 10MediaWiki-extensions-EducationProgram, 10Repository-Admins, 10Patch-For-Review: Archive EducationProgram - https://phabricator.wikimedia.org/T214457 (10MacFan4000) Remaining to items will need to be done by somebody with proper access. [10:50:24] (03CR) 10Hashar: [C: 04-1] "> > The master node has been disabled." [integration/config] - 10https://gerrit.wikimedia.org/r/577678 (https://phabricator.wikimedia.org/T247109) (owner: 10Dduvall) [11:06:41] (03PS1) 10Ema: Test atskafka with debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/578498 (https://phabricator.wikimedia.org/T237993) [11:17:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Lucas_Werkmeister_WMDE) The beta cluster seems to get brokener and brokener. On en beta, the [main page](https://e... [11:41:57] 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Dvorapa) >>! In T247078#5955700, @Lucas_Werkmeister_WMDE wrote: > The beta cluster seems to get brokener and broke... [11:43:19] 10Gerrit, 10Release-Engineering-Team-TODO, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Lucas_Werkmeister_WMDE) FWIW, I’ve just encountered this for the third time in the... [11:44:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Lucas_Werkmeister_WMDE) No, in the original state of this task the main page sidebar link was broken as well, at l... [11:51:30] 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Dvorapa) >>! In T247078#5955776, @Lucas_Werkmeister_WMDE wrote: > No, in the original state of this task the main... [11:56:51] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10nshahquinn-wmf) @Aklapper, I would like the ability to create projects to help organize the work of my team (WMF Product Analytics) without needing to wait for help. Currently, no on... [12:00:17] (03CR) 10Hashar: [C: 04-1] "> > If the issue is the # of executors, simply raise the number. 3 is an arbitrary value, it can be raise to eg 12 ( https://integration." [integration/config] - 10https://gerrit.wikimedia.org/r/577678 (https://phabricator.wikimedia.org/T247109) (owner: 10Dduvall) [12:08:36] (03PS1) 10Lars Wirzenius: fix build of Debian package on jessie (CI) [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 [12:09:28] (03CR) 10Lars Wirzenius: "Please review this change to fix building of Debian package in jessie (read: our CI). I've tested the build locally in a jessie environmen" [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 (owner: 10Lars Wirzenius) [12:19:05] 10Gerrit, 10Release-Engineering-Team-TODO, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Lucas_Werkmeister_WMDE) p:05High→03Unbreak! So far today, I’ve seen more gate-... [12:34:11] 10Project-Admins: Lingua Libre Milestone - https://phabricator.wikimedia.org/T246992 (10Eavqwiki) [12:54:19] 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) > The metric won't be a single one for all services though, when Prometheus pulls from k8s servi... [13:33:42] 10Continuous-Integration-Infrastructure, 10Cleanup, 10MediaWiki-extensions-EducationProgram, 10Repository-Admins, 10Patch-For-Review: Archive EducationProgram - https://phabricator.wikimedia.org/T214457 (10hashar) [13:35:40] 10Continuous-Integration-Infrastructure, 10Cleanup, 10MediaWiki-extensions-EducationProgram, 10Repository-Admins, 10Patch-For-Review: Archive EducationProgram - https://phabricator.wikimedia.org/T214457 (10hashar) 05Open→03Resolved I have deleted the two remaining pieces: * https://doc.wikimedia.org... [13:52:55] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24 - https://phabricator.wikimedia.org/T237191 (10Count_Count) I think we should call [[ https://github.com/wikimedia/mediawiki-extensions-FlaggedRevs/blob... [14:01:23] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24 - https://phabricator.wikimedia.org/T237191 (10Reedy) >>! In T237191#5956127, @Count_Count wrote: > I think we should call [[ https://github.com/wikimed... [14:04:50] (03CR) 10Thcipriani: [C: 03+1] "Looks good to me! One inline "nit"." (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 (owner: 10Lars Wirzenius) [14:09:31] (03CR) 10Thcipriani: WIP: Scap3 deploy repo for zuul (031 comment) [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (owner: 1020after4) [14:13:30] (03PS2) 10Lars Wirzenius: fix build of Debian package on jessie (CI) [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 [14:14:00] (03CR) 10Lars Wirzenius: fix build of Debian package on jessie (CI) (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 (owner: 10Lars Wirzenius) [14:14:34] (03CR) 10Lars Wirzenius: [C: 03+2] "Since Tyler gave a conditional +2, and I've fulfilled the condition, I'm doing the +2 myself now." [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 (owner: 10Lars Wirzenius) [14:16:04] (03Merged) 10jenkins-bot: fix build of Debian package on jessie (CI) [tools/scap] - 10https://gerrit.wikimedia.org/r/578510 (owner: 10Lars Wirzenius) [14:27:51] 10Project-Admins, 10Multimedia, 10Structured-Data-Backlog: Clarify confusion between Multimedia and Structured Data team and update docs accordingly - https://phabricator.wikimedia.org/T240281 (10dr0ptp4kt) Hi team, we've reviewed this, and here's the guidance: The Structured Data engineering team should pl... [14:56:14] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24 - https://phabricator.wikimedia.org/T237191 (10Count_Count) > How does that help 3rd party wikis? ... Not at all. Why is the config for the WMF-Wikis... [14:57:00] (03CR) 10Jforrester: [C: 03+2] Test atskafka with debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/578498 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [14:57:49] (03Merged) 10jenkins-bot: Test atskafka with debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/578498 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [14:58:41] !log Add CI for operations/software/atskafka T237993 [14:58:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:58:45] T237993: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 [15:01:22] 10Project-Admins, 10Collection, 10OCG-General: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312 (10Aklapper) Big thanks to Tgr for taking the time to explain the stack! I edited the project descriptions of #proton, #PDF-Rendering and #Electron-PDFs accordingly. [15:05:23] (03PS1) 10Ema: atskafka: use -backports [integration/config] - 10https://gerrit.wikimedia.org/r/578539 (https://phabricator.wikimedia.org/T237993) [15:07:24] 10Project-Admins, 10Collection, 10OCG-General: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312 (10Aklapper) >>! In T161312#3850230, @Jdlrobson wrote: > @ovasileva would be most appropriate to help with deciding what to do here ^ @ovasileva: Excluding tasks wi... [15:07:33] 10Project-Admins: Please create project tag 'onfire' (or 'sre-onfire') - https://phabricator.wikimedia.org/T247313 (10CDanis) [15:07:44] (03CR) 10Jforrester: [C: 03+2] atskafka: use -backports [integration/config] - 10https://gerrit.wikimedia.org/r/578539 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [15:08:29] (03Merged) 10jenkins-bot: atskafka: use -backports [integration/config] - 10https://gerrit.wikimedia.org/r/578539 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [15:14:59] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Aklapper) [15:23:03] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Reedy) >>! In T237191#5956388, @Count_Count wrote: > Why is the config for the WMF-Wikis in flagged... [15:24:32] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic: debian-glue-backports not enabling backports on buster - https://phabricator.wikimedia.org/T247316 (10ema) [15:25:43] (03CR) 10Hashar: WIP: Scap3 deploy repo for zuul (032 comments) [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (owner: 1020after4) [15:29:57] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic, 10Patch-For-Review: debian-glue-backports not enabling backports on buster - https://phabricator.wikimedia.org/T247316 (10ema) p:05Triage→03Medium [15:39:06] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid, 10User-Ryasmeen: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10Jdforrester-WMF) ` jforrester@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:8001 http://en.wi... [15:59:57] (03CR) 1020after4: "> Patch Set 2:" (033 comments) [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (owner: 1020after4) [16:04:26] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Order of magnitude for New CI hosting budget - https://phabricator.wikimedia.org/T247320 (10thcipriani) [16:07:11] 10Project-Admins: Please create project tag 'onfire' (or 'sre-onfire') - https://phabricator.wikimedia.org/T247313 (10Jdforrester-WMF) #sre-onfire Is probably better. [16:08:13] 10Project-Admins: Please create project tag 'onfire' (or 'sre-onfire') - https://phabricator.wikimedia.org/T247313 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Created. [17:04:46] (03PS3) 1020after4: Scap3 deploy repo for zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 [17:24:02] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10thcipriani) [17:29:06] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10thcipriani) Strangely, I'm not seeing the graphs in gerr... [17:47:20] 10Scap, 10MediaWiki-Debug-Logger, 10Performance-Team, 10observability, and 3 others: Standardise on Logstash channel for fatal exceptions (fatal, exception, DeferredUpdates) - https://phabricator.wikimedia.org/T247113 (10Krinkle) p:05Triage→03Medium [17:47:47] 10Scap, 10MediaWiki-Debug-Logger, 10Performance-Team, 10observability, and 3 others: Standardise on Logstash channel for fatal exceptions (fatal, exception, DeferredUpdates) - https://phabricator.wikimedia.org/T247113 (10Krinkle) a:03Krinkle [17:48:23] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid, 10User-Ryasmeen: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10ssastry) This worked for me. I see that you missed the ".beta." in your urls ` ssastry@deployment-parsoid11:~$ curl -x deployment-... [17:50:31] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid, 10User-Ryasmeen: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10ssastry) But, when I open VE (which open successfully probably because the page is cached in RB), and click review changes after a... [17:53:33] 10Project-Admins: Create a wmfdata-python project - https://phabricator.wikimedia.org/T247060 (10Aklapper) Hi, is that code location going to be changed (either to move under the `wikimedia` organization on Github, or to Wikimedia Gerrit)? [17:54:47] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10thcipriani) Looking specifically for upload-pack errors... [17:55:57] 10Project-Admins: Create a wmfdata-python project - https://phabricator.wikimedia.org/T247060 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project has been created: #wmfdata-python Please encourage interested people to visit the project and to join the project as {icon users} members, and... [17:58:50] 10Project-Admins: Lingua Libre Sign It - https://phabricator.wikimedia.org/T246993 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #signit as been created: https://phabricator.wikimedia.org/project/view/4628/ Please encourage interested people to visit the project and to join the pro... [17:59:50] 10Project-Admins: Lingua Libre Milestone - https://phabricator.wikimedia.org/T246992 (10Aklapper) 05Open→03Resolved a:03Aklapper Sure, created now! See https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milestones for more information about milestones. [18:35:12] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Zoranzoki21, @nshahquinn-wmf: Hi, I have added you both. (Usual default disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_rena... [18:46:59] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Zoranzoki21) >>! In T706#5957478, @Aklapper wrote: > @Zoranzoki21, @nshahquinn-wmf: Hi, I have added you both. (Usual default disclaimer: Please follow [guidelines](https://www.media... [18:56:14] (03PS1) 1020after4: check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 [18:56:37] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10thcipriani) p:05Unbreak!→03High Lowering priority si... [18:56:39] (03CR) 10jerkins-bot: [V: 04-1] check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [18:57:27] (03CR) 10Brennen Bearnes: [C: 03+1] "This looks good - I'll give it an actual test after train window." [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [18:57:59] (03PS2) 1020after4: check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 [19:02:18] (03CR) 10Lars Wirzenius: "Once tests pass, treat this +1 as a +2. If you want to make the env check, it doesn't need another review I think. Or just +2 yourself." (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [19:03:55] (03PS3) 1020after4: check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 [19:06:31] (03CR) 10Lars Wirzenius: [C: 03+2] check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [19:06:59] (03Merged) 10jenkins-bot: check for SSH_AUTH_SOCK before running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [19:07:22] (03CR) 1020after4: check for SSH_AUTH_SOCK before running branch.py (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/578584 (owner: 1020after4) [19:55:10] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Dzahn) We noticed that /var/log/syslog on gerrit1001 was... [19:57:35] (03CR) 1020after4: Scap3 deploy repo for zuul (032 comments) [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (owner: 1020after4) [20:07:03] 10Phabricator, 10Security Preview, 10Security Readiness Reviews, 10Security-Team, 10Security: combine security readiness review and security preview boards with third tag - https://phabricator.wikimedia.org/T247326 (10chasemp) [20:08:54] 10Phabricator, 10Security Preview, 10Security Readiness Reviews, 10Security-Team, and 2 others: combine security readiness review and security preview boards with third tag - https://phabricator.wikimedia.org/T247326 (10chasemp) {H357} [20:16:58] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Puppet fails on Beta Cluster because Safe_service_restart is expecting lvs_services set though `has_lvs: false` - https://phabricator.wikimedia.org/T247151 (10Dzahn) >>! In T247151#5950705, @Krenair wrote: > Remaining err... [20:24:03] 10Phabricator, 10Security Preview, 10Security Readiness Reviews, 10Security-Team, and 2 others: combine security readiness review and security preview boards with third tag - https://phabricator.wikimedia.org/T247326 (10chasemp) https://phabricator.wikimedia.org/tag/security_reviews_team/ [20:34:26] 10Phabricator, 10Security Preview, 10Security Readiness Reviews, 10Security-Team, and 2 others: combine security readiness review and security preview boards with third tag - https://phabricator.wikimedia.org/T247326 (10chasemp) 05Open→03Resolved [20:52:41] 10Phabricator, 10Project-Admins, 10Security Preview, 10Security Readiness Reviews, and 3 others: combine security readiness review and security preview boards with third tag - https://phabricator.wikimedia.org/T247326 (10DannyS712) [21:04:48] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Puppet fails on Beta Cluster because Safe_service_restart is expecting lvs_services set though `has_lvs: false` - https://phabricator.wikimedia.org/T247151 (10Krenair) 05Open→03Resolved a:03Krenair @Jdforrester-WMF... [21:04:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Parsoid, 10Patch-For-Review: Replace deployment-mediawiki-parsoid10 with a "purer" deployment-parsoid11 box - https://phabricator.wikimedia.org/T246854 (10Krenair) [21:26:18] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Puppet fails on Beta Cluster because Safe_service_restart is expecting lvs_services set though `has_lvs: false` - https://phabricator.wikimedia.org/T247151 (10Jdforrester-WMF) I'd already set it in horizon this morning to... [21:35:23] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10releng-201516-q3, and 2 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908 (10Krinkle) [21:39:39] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Move scap-vagrant to Gerrit - https://phabricator.wikimedia.org/T216483 (10MarcoAurelio) @thcipriani It's been a year since the last ping :-) Do you object moving this to Gerrit? If not, which repo... [22:18:17] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T233871 (10brennen) Group0 is currently on wmf.23. `scap sync "testwiki to php-1.35.0-wmf.23 and rebuild l10n cache"` took a long... [23:21:01] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Krinkle) Still seen 30min ago at [change 578402](https:/... [23:52:29] !log Restarting jenkins for upgrade. [23:52:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL