[00:02:32] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) [00:02:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Work out how we are going to cut deploy branches without gerrit HTTP Token - https://phabricator.wikimedia.org/T218597 (10zeljkofilipin) 05Open→03Resolved a:05zeljkofilipin→03thcipriani Assigning to @thcipria... [00:05:06] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10VisualEditor, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10User-zeljkofilipin: fatal: Project not found: p/VisualEditor/VisualEditor - https://phabricator.wikimedia.org/T218694 (10zeljkofilipin) >>! In T218694#5037194, @Krinkle wrote: >... [00:07:08] 10Gerrit, 10Release-Engineering-Team: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Legoktm) [00:07:11] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) [00:07:16] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10VisualEditor, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), and 2 others: fatal: Project not found: p/VisualEditor/VisualEditor - https://phabricator.wikimedia.org/T218694 (10zeljkofilipin) 05Open→03Resolved a:05zeljkofilipin→03Jdforre... [00:08:51] 10Gerrit: Create Gerrit Administrator right policy - https://phabricator.wikimedia.org/T218686 (10Legoktm) Can we use the normal server access policy for this? If you're trusted and have a demonstrable need, you can ask for and receive Gerrit administrator access? [00:09:04] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) The branch is finally cut. Thanks to @thcipriani for all the help. There are no more blockers. It's too late to continue... [00:18:39] (03PS1) 10Legoktm: Upgrade PHP_CodeSniffer to 3.4.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/497664 [00:57:42] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10phan: mwext-php70-phan-docker doesn't use composer cache - https://phabricator.wikimedia.org/T217479 (10Legoktm) Hmm, doesn't seem to be working. @hashar any ideas what's wrong? [02:55:07] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:02:22] PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:39:43] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Krinkle) [03:46:33] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Legoktm) Will the current plugin continue to work properly with ast 0.1.4? Or do we also need to keep 0.1.2 for back/compat reasons? [03:47:21] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Krinkle) Last quarter I've put the basics in place for Node 10 and npm 6. This was mainly for #Fresnel, but I took t... [05:00:58] Project beta-scap-eqiad build #241998: 04FAILURE in 8 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241998/ [05:12:01] Yippee, build fixed! [05:12:02] Project beta-scap-eqiad build #241999: 09FIXED in 9 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241999/ [05:39:54] (03CR) 10MaxSem: [C: 03+2] Upgrade PHP_CodeSniffer to 3.4.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/497664 (owner: 10Legoktm) [05:40:47] (03Merged) 10jenkins-bot: Upgrade PHP_CodeSniffer to 3.4.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/497664 (owner: 10Legoktm) [05:41:45] ty MaxSem :) [05:41:57] yw [07:24:01] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Daimona) @Legoktm I think the current version should work with 0.1.4. But actually, could you please add 0.1.5 as well? We'll need it for the next upgrade. Thanks! [07:27:26] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Legoktm) >>! In T218719#5039189, @Daimona wrote: > @Legoktm I think the current version should work with 0.1.4. Gotcha. I can do that then. > But actually, could you please add... [07:32:50] (03PS1) 10Giuseppe Lavagetto: s/Trusted Users/Trusted Contributors/ [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497694 [07:34:13] (03CR) 10Marostegui: [C: 03+1] s/Trusted Users/Trusted Contributors/ [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497694 (owner: 10Giuseppe Lavagetto) [07:34:31] (03CR) 10Addshore: [C: 03+1] s/Trusted Users/Trusted Contributors/ [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497694 (owner: 10Giuseppe Lavagetto) [07:36:07] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Daimona) @Legoktm Cool, thanks! I think for now you can use whatever hack comes more handy for seccheck, as at the end of the upgrade we should ideally have a single version insta... [07:37:18] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Legoktm) From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once... [07:44:57] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade ext-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Daimona) @Legoktm Right. But anyway at the end of the migration we'll only have 0.1.4 for the current version, and ?[0] for the final version based on 1.2.x. [0]: To be determine... [08:11:08] (03Abandoned) 10Giuseppe Lavagetto: s/Trusted Users/Trusted Contributors/ [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497694 (owner: 10Giuseppe Lavagetto) [08:27:10] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade php-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10hashar) [08:34:25] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade php-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10hashar) Tentatively that has been solved already? We have T174339 to upgrade to phan 0.8.5, which requires a newer version of php-ast than the one we had. {T174338} was the task... [08:36:58] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade php-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10hashar) > From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once... [08:42:43] 10Release-Engineering-Team (Kanban), 10Zuul: Evaluate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10LarsWirzenius) I've used Ansible for several years now. In some ways I like it a lot, especially the "push" model, which doesn't require an agent running constantly on the target. However, I don't like... [08:44:59] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade php-ast to 0.1.4 - https://phabricator.wikimedia.org/T218719 (10Daimona) >>! In T218719#5039252, @hashar wrote: > We have T174339 to upgrade to phan 0.8.5, which requires a newer version of php-ast than the one we had. IIUC that's for the pha... [09:02:07] 10Continuous-Integration-Config, 10ArticleFeedbackv5: Add CI to mediawiki/extensions/ArticleFeedbackv5 - https://phabricator.wikimedia.org/T218759 (10hashar) [09:35:40] 10Gerrit: Create Gerrit Administrator right policy - https://phabricator.wikimedia.org/T218686 (10hashar) We need a dedicated policy. Granting Gerrit administrative rights is a lot more responsibilities and require a lot of trust. It is the same level as granting rights to the puppet repository. [10:34:29] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [10:46:56] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10zeljkofilipin) [10:46:59] 10Release-Engineering-Team (Kanban): Evaluate Tekton Pipeline - https://phabricator.wikimedia.org/T217912 (10zeljkofilipin) 05Declined→03Open @dduvall said in IRC that he would like to evaluate it. He was able to get it running locally. [10:59:45] (03CR) 10Addshore: [C: 03+1] Add WikibaseMediaInfo to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/480463 (https://phabricator.wikimedia.org/T204153) (owner: 10Jforrester) [10:59:54] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:04:04] Project mediawiki-core-doxygen-docker build #5553: 04FAILURE in 0.31 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5553/ [11:04:57] Project beta-code-update-eqiad build #239495: 04FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239495/ [11:08:48] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [11:08:59] 10Release-Engineering-Team (Kanban): Evaluate Tekton Pipeline - https://phabricator.wikimedia.org/T217912 (10zeljkofilipin) See P8237. [11:10:53] 10Gerrit, 10Developer-Advocacy, 10Documentation: Update Wikimedia's Gerrit documentation for new UI (GWTUI to be superseded by polygerrit) - https://phabricator.wikimedia.org/T179759 (10Aklapper) [11:14:19] Yippee, build fixed! [11:14:20] Project beta-code-update-eqiad build #239496: 09FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239496/ [11:24:12] 10Release-Engineering-Team (Watching / External), 10Operations, 10puppet-compiler, 10Puppet: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066 (10hashar) We have a Jenkins job T97513 which has been made to recognizes `Hosts:` in commit message to passe a li... [11:40:48] (03PS1) 10Paladox: Block “Registered Users” on refs/* [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497752 [11:41:41] (03PS2) 10Paladox: Block “Registered Users” on refs/* [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497752 [11:42:35] (03CR) 10Paladox: [V: 03+1] "Tested locally on https://gerrit.git.wmflabs.org/r/" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497752 (owner: 10Paladox) [11:55:25] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [12:01:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10jbond) patch is merged, let me know if there is anything elses from my side [12:15:01] Yippee, build fixed! [12:15:02] Project mediawiki-core-doxygen-docker build #5554: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5554/ [12:28:38] question about diffusion mirrors of gerrit repos. are "Initialization Error"s because of gerrit outages on anyones radar? e.g. https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ [13:19:21] Pablo_WMDE: it probably happened when Gerrit was disabled yes [13:20:21] !log Scheduled update of Diffusion repository wikibase-termbox via https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ on request of Pablo_WMDE [13:20:37] hashar: Failed to log message to wiki. Somebody should check the error logs. [13:20:37] I don't quite know where the logs are though [13:20:43] and it has a 30 seconds update frequency [13:20:52] so maybe Diffusion can not reach gerrit :/ [13:21:11] which probably mean we need a task [13:22:20] Pablo_WMDE: I am not familiar with Diffusion unfortunately :( [13:23:09] It appears none of the repos have updated hashar [13:23:29] it still shows an error on https://phabricator.wikimedia.org/source/wikibase-termbox/manage/ [13:23:43] * Pablo_WMDE back [13:25:26] maybe scheduling an update actually fixed it [13:28:06] hi hashar. thanks for looking into it. the 30s don't seems to work in our favor and, if i understand you correctly, the manual update (i don't have permission to do so) did not yield anything positive yet. which phab tag do you suppose would be a good one to use for a ticket? [13:28:41] Even Gerrit repo from upstream is not updating in diffusion, so not related to last nights update [13:28:46] paladox: #phabricator I guess [13:28:50] err [13:28:51] More like phd [13:29:00] paladox: #phabricator I guess and poke me with the task ticket [13:29:09] #Repository-Admins maybe? [13:29:19] yeah that one as well probably [13:29:22] hashar: ok, though I have to go now :( [13:29:32] and subscribe twentyafterfour to it (I cant remember his exact name on phabricator) [13:29:36] So doint have time to file one atm [13:29:41] paladox: don't worry! Enjoy your afternoon :] [13:29:47] hashar: mmodell is the one :) [13:29:50] Thanks :) [13:29:57] paladox: you are awesome. Thank you [13:30:08] Pablo_WMDE: also https://phabricator.wikimedia.org/source/wikibase-termbox/manage/basics/ shows some Initialization error and some failing command [13:30:16] namely a timeout reaching the address [13:30:32] but I don't have access to the server or daemon logs :/ [13:31:55] i saw this error, came here because i hoped there would be ppl with access (or potentially already working on such problems in general, which would be understandable after the repeated gerrit downtime/changes) [13:32:18] thanks for your time, hashar. I'll poke on phab [13:43:07] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Have CI run seccheck tests - https://phabricator.wikimedia.org/T218721 (10Bawolff) FYI, these tests should already be run via CI (as part of composer tests) [13:47:29] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Have CI run seccheck tests - https://phabricator.wikimedia.org/T218721 (10Daimona) Ah, now I see. Would it still benefit from phpunit? [13:50:48] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10zeljkofilipin) [13:51:01] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) [13:51:04] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10zeljkofilipin) [13:51:10] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10zeljkofilipin) p:05Triage→03Unbreak! [13:54:48] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10hashar) 13:44:00 Started prune-git-branches Received disconnect from 2620:0:861:3:208:80:154:85 port 29418:2: Too many authentication failures: 7 Authentication failed. `p... [13:55:34] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Pablo-WMDE) > The question is basically: how can a team tasked with everything else it is doing (deployments, tooling, CI, etc) keep up with an SRE team of 20 peop... [13:58:23] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate Spinnaker - https://phabricator.wikimedia.org/T218335 (10zeljkofilipin) a:05zeljkofilipin→03None [14:04:17] 10Diffusion, 10Repository-Admins, 10Wikidata: Diffusion: wikibase-termbox gerrit mirror connection time out - https://phabricator.wikimedia.org/T218780 (10Mainframe98) This is happening with other repositories too, such as rEOTP, rSVEC, and rMW. [14:05:48] Project beta-scap-eqiad build #242046: 04FAILURE in 9 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242046/ [14:14:57] 10Release-Engineering-Team (Next), 10ChangeProp, 10Release Pipeline, 10serviceops, and 2 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10Pchelolo) [14:16:15] 10Release-Engineering-Team (Next), 10ChangeProp, 10Release Pipeline, 10serviceops, and 2 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10Pchelolo) [14:16:53] Yippee, build fixed! [14:16:53] Project beta-scap-eqiad build #242047: 09FIXED in 9 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242047/ [14:45:57] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Block “Registered Users” on refs/* [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497752 (owner: 10Paladox) [14:46:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) Train is blocked on T218783. [14:54:56] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10Reedy) [14:59:42] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10hashar) When scap clean is issued, on Gerrit sshd_log we eventually see: AUTH FAILURE FROM 2620:0:861:103:10:64:32:16 no-matching-key But `ssh -p 29418 zfil... [15:17:57] 10Continuous-Integration-Config: post-merge jenkins job not run after merge for maven based search projects - https://phabricator.wikimedia.org/T218550 (10Gehel) [15:19:30] zeljkof: hey, are you doing the train this week? [15:20:13] mobrovac: yes [15:20:30] ok so it's happening in the EU time [15:20:34] kk thnx [15:20:39] with not much luck so far, only the branch has been cut [15:20:42] https://phabricator.wikimedia.org/T206676 [15:20:47] ooh [15:20:59] mobrovac: it's likely I'll have to do some deployment in the US window [15:21:02] zeljkof: can you tell me when exactly has the .22 branch been cut? [15:21:27] since the EU is done, and we're still not even at group 0 [15:21:53] mobrovac: around 1am today (utc+1) https://phabricator.wikimedia.org/T206676#5038897 [15:41:50] (03PS1) 10D3r1ck01: Re-enable CI tests for ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/497792 (https://phabricator.wikimedia.org/T218759) [15:43:16] (03CR) 10jerkins-bot: [V: 04-1] Re-enable CI tests for ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/497792 (https://phabricator.wikimedia.org/T218759) (owner: 10D3r1ck01) [15:46:32] (03PS2) 10D3r1ck01: Re-enable CI tests for ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/497792 (https://phabricator.wikimedia.org/T218759) [15:51:10] PROBLEM - Puppet staleness on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:54:27] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0] [15:58:23] 10Continuous-Integration-Config, 10ArticleFeedbackv5, 10Patch-For-Review, 10User-D3r1ck01: Add CI to mediawiki/extensions/ArticleFeedbackv5 - https://phabricator.wikimedia.org/T218759 (10D3r1ck01) a:03D3r1ck01 [15:59:10] (03CR) 10D3r1ck01: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/497792 (https://phabricator.wikimedia.org/T218759) (owner: 10D3r1ck01) [16:02:27] (03PS1) 10MacFan4000: Archive Configure ext [integration/config] - 10https://gerrit.wikimedia.org/r/497800 (https://phabricator.wikimedia.org/T185227) [16:03:16] (03PS2) 10MacFan4000: Archive Configure ext [integration/config] - 10https://gerrit.wikimedia.org/r/497800 (https://phabricator.wikimedia.org/T185227) [16:08:58] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10dduvall) [16:09:02] 10Release-Engineering-Team (Kanban), 10Jenkins: Evaluate Jenkins X - https://phabricator.wikimedia.org/T218334 (10dduvall) 05Open→03Resolved I followed the [[https://jenkins-x.io/getting-started/create-cluster/ | Getting Started]] section of the Jenkins X documentation to get it installed locally using min... [16:13:40] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10greg) p:05Unbreak!→03High Not a blocker to the train. But needs to be dealt with this week or next. [16:13:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10greg) [16:13:54] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10greg) [16:14:22] 10MediaWiki-Codesniffer: Code sniffer expects mismatched syntax for referenced varargs @params - https://phabricator.wikimedia.org/T218797 (10Anomie) [16:16:34] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Xqt) [16:17:31] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Xqt) [16:21:27] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Aklapper) The proper solution would be to move most tags of H6 to become [subprojects](https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milestones) of Pyw... [16:28:05] PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5) [16:31:24] thcipriani: around [16:31:25] ? [16:31:35] I have some q's [16:32:50] But I'll just ask here and maybe someone else can answer [16:35:10] Jenkins: I want to modify the UI to expose some parameters [16:36:55] It would be good to have an existing jjb to look at [16:40:01] greg-g: anyone I should talk to in particular? [16:40:54] hexmode: right now we're all dealing with the gerrit and phab vandalism so our response to non-UBN! things will be delayed. tyler, for instance, was up until 1:30am his time dealing with things. [16:41:22] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Xqt) As I can see all of the projects ending with .py are script releated subprojects and attributed as `Component`; they are exclusive. The other are marked as `Tag` and should also be usable with th... [16:41:25] hexmode: maybe look through jjb jobs for what you're looking for? [16:41:34] greg-g: oh, sorry. I saw the vandalism and didn't realize it was ongoing. [16:41:36] will do [16:45:08] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Xqt) Seems this is similar to T170090. [16:51:08] Project beta-scap-eqiad build #242061: 04FAILURE in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242061/ [16:57:36] (03CR) 10Jforrester: [C: 03+1] WikibaseLexemeCirrusSearch doesn't have browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/497600 (owner: 10Smalyshev) [17:02:08] Yippee, build fixed! [17:02:09] Project beta-scap-eqiad build #242062: 09FIXED in 9 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242062/ [18:38:36] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate Spinnaker - https://phabricator.wikimedia.org/T218335 (10brennen) a:03brennen [18:55:16] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [18:57:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) {P8247} [18:58:54] uh [18:58:58] clearly this is not good [18:59:06] I think designate might be upset [19:00:23] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [19:00:28] yeah so [19:00:48] designate saw the TXT entries and stopped serving the *.beta.wmflabs.org A record. WTF. [19:01:44] * Krenair tests [19:05:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) (That said, in the process I think I may have stumbled upon a rather serious bug in the cloud DNS infrastructure... [19:07:44] does a `_acme-challenge.wikipedia.beta.wmflabs.org` TXT record really break `8*.beta.wmflabs.org` A record responses for `en.wikipedia.beta.wmflabs.org`? :| [19:32:05] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Re-enable CI tests for ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/497792 (https://phabricator.wikimedia.org/T218759) (owner: 10D3r1ck01) [19:39:38] 10Release-Engineering-Team (Kanban): Evaluate Spinnaker - https://phabricator.wikimedia.org/T218335 (10zeljkofilipin) [19:41:43] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants null even if type is nullable - https://phabricator.wikimedia.org/T218816 (10cscott) [19:42:06] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants redundant "mixed|null" - https://phabricator.wikimedia.org/T218324 (10cscott) See also {T218816}. [19:44:19] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) Group0 at 1.33.0-wmf.22 ([[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/497765 | gerrit ]], [[ https:... [19:47:21] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants null even if type is nullable - https://phabricator.wikimedia.org/T218816 (10Anomie) `?string` and `string|null` mean the same thing, I suppose the question is which of those we want in the docs. [20:05:33] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) [20:17:27] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants null even if type is nullable - https://phabricator.wikimedia.org/T218816 (10thiemowmde) Call me oldskool, but I find `string|null` much more readable. The question mark not only looks like a misplaced character, it can also... [20:28:42] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) Group1 at 1.33.0-wmf.22 ([[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/497859 | gerrit ]], [[ https:... [20:56:28] Project beta-scap-eqiad build #242083: 04FAILURE in 9 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242083/ [21:05:24] PROBLEM - Host deployment-sessionstore01 is DOWN: CRITICAL - Host Unreachable (172.16.3.4) [21:07:05] 20:56:27 sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild on deployment-mwmaint01.deployment-prep.eqiad.wmflabs returned [255]: sign_and_send_pubkey: signing failed: communication with agent failed [21:07:05] 20:56:27 Permission denied (publickey). [21:07:06] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Dvorapa) >>! In T218798#5040699, @Aklapper wrote: > The proper solution would be to move most tags of H6 to become [subprojects](https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Pr... [21:07:55] Project beta-scap-eqiad build #242084: 04STILL FAILING in 8 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242084/ [21:09:00] ^ missing extension, jenkins said [21:11:37] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Dvorapa) I take care of adding #pywikibot to new pywikibot tasks since April 2017 I think. This Harold rule would just do the job I have done until now (and also back in time) automatically, therefore... [21:15:07] 10Phabricator, 10Pywikibot: Add a new Herald rule for pywikibot - https://phabricator.wikimedia.org/T218798 (10Dvorapa) BTW there are cca [[https://phabricator.wikimedia.org/maniphest/query/Al15sXKb0Qme/|169]] tasks, that are missing the Pywikibot tag [21:15:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10zeljkofilipin) [21:17:50] Project beta-scap-eqiad build #242085: 04STILL FAILING in 8 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242085/ [21:18:31] hauskatze, well I can `SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mwmaint01` [21:19:23] Krenair: that was the jenkins complain - now it says that an extension is missing [21:19:41] code-update-eqiad seems to work okay, but beta-scap-eqiad not [21:24:59] twentyafterfour: hello - https://phabricator.wikimedia.org/diffusion/TMAB/manage/ seems to be stuck in the "prioritized" status and it's out of sync since yesterday. Too much backlog? [21:25:41] hauskatze: sync is currently disabled until after tonight's phabricator deployment [21:26:50] twentyafterfour: aah, than explains - will tonights phab deployment import an upstream i18n fix I reported last week? https://phabricator.wikimedia.org/T218447 [21:27:24] hauskatze: if the patch landed upstream then yes it should be deployed tonight [21:27:33] if it's not on upstream/stable then I'll need to cherry-pick it [21:27:45] hauskatze: can you link me to the patch? [21:27:45] Project beta-scap-eqiad build #242086: 04STILL FAILING in 8 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242086/ [21:27:48] oh you did [21:28:07] sorry bad at reading today [21:28:20] evan did https://secure.phabricator.com/D20290 and it's merged but I don't understand differential much [21:32:05] 10Release-Engineering-Team (Kanban): Evaluate ArgoCD - https://phabricator.wikimedia.org/T218827 (10dduvall) [21:37:40] Project beta-scap-eqiad build #242087: 04STILL FAILING in 8 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242087/ [21:48:12] Project beta-scap-eqiad build #242088: 04STILL FAILING in 8 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242088/ [21:58:09] Project beta-scap-eqiad build #242089: 04STILL FAILING in 8 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242089/ [22:08:06] Project beta-scap-eqiad build #242090: 04STILL FAILING in 8 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242090/ [22:14:56] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/CrawlableAllPages] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497933 [22:14:58] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/CrawlableAllPages] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497933 (owner: 10QChris) [22:15:19] (03PS1) 10QChris: Import done. Revoke import grants [extensions/CrawlableAllPages] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497935 [22:15:21] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/CrawlableAllPages] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497935 (owner: 10QChris) [22:16:34] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants null even if type is nullable - https://phabricator.wikimedia.org/T218816 (10cscott) Because the type hint just below it will say ?string not string|null? We should be using the one markup language which is used by PHP, not... [22:16:44] herron, hi, is the ores uncommitted change on deployment-puppetmaster03 yours? [22:16:59] hey, yeah probably mine and can be removed if needed [22:17:41] it's adding "logstash_host: localhost" to hieradata/role/common/ores.yaml [22:18:06] Project beta-scap-eqiad build #242091: 04STILL FAILING in 8 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242091/ [22:20:37] I notice beta scap fails and then fixes itself constantly why is that? (Just curious) [22:20:57] Project beta-update-databases-eqiad build #32569: 04FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32569/ [22:24:19] (I've stashed that change while I muck about with the repo, will stash pop later if I remember) [22:25:17] 10Continuous-Integration-Infrastructure, 10Phabricator (Upstream), 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10hashar) From: Paolo Greppi Date: Wed, 20 Mar 2019 18:52:04 +0100 Hi Antoine, an... [22:26:15] 10Continuous-Integration-Infrastructure, 10Phabricator (Upstream), 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10hashar) From: Antoine Musso Date: Wed, 20 Mar 2019, 22:24 Hello Paolo, Thank you... [22:28:04] Project beta-scap-eqiad build #242092: 04STILL FAILING in 8 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242092/ [22:29:45] 10Continuous-Integration-Infrastructure, 10Phabricator (Upstream), 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10hashar) >>! In T218233#5036593, @Krinkle wrote: > Looks like all the Debian channe... [22:37:13] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: connect to address en.wikipedia.beta.wmflabs.org and port 443: Connection refused [22:38:01] Project beta-scap-eqiad build #242093: 04STILL FAILING in 8 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242093/ [22:42:14] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48767 bytes in 0.875 second response time [22:48:07] Project beta-scap-eqiad build #242094: 04STILL FAILING in 8 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242094/ [22:48:07] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10Bawolff) [22:51:15] 10Release-Engineering-Team (Kanban), 10Jenkins: Evaluate Jenkins X - https://phabricator.wikimedia.org/T218334 (10hashar) Excellent report Dan. I really envy your excellent sense of humor! [22:57:27] 10Beta-Cluster-Infrastructure: Beta eswikibooks certificate issues - https://phabricator.wikimedia.org/T199387 (10Krenair) works now with some puppet cherry-picks :) [22:57:59] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10HTTPS: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Krenair) works now with some puppet cherry-picks [22:58:06] Project beta-scap-eqiad build #242095: 04STILL FAILING in 8 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242095/ [23:06:15] (03CR) 10EBernhardson: [C: 03+1] WikibaseLexemeCirrusSearch doesn't have browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/497600 (owner: 10Smalyshev) [23:07:53] Project beta-scap-eqiad build #242096: 04STILL FAILING in 8 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242096/ [23:17:54] Project beta-scap-eqiad build #242097: 04STILL FAILING in 8 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242097/ [23:21:39] Yippee, build fixed! [23:21:40] Project beta-update-databases-eqiad build #32570: 09FIXED in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32570/ [23:28:03] Project beta-scap-eqiad build #242098: 04STILL FAILING in 8 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242098/ [23:31:20] could someone hit +2 on https://gerrit.wikimedia.org/r/c/integration/config/+/497600? [23:31:46] it's blocking merge for a patch because I stupidly told CI it has tests that it actually doesn't... [23:37:49] Project beta-scap-eqiad build #242099: 04STILL FAILING in 8 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242099/ [23:47:48] Project beta-scap-eqiad build #242100: 04STILL FAILING in 8 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242100/ [23:50:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10Jdforrester-WMF) [23:57:45] Project beta-scap-eqiad build #242101: 04STILL FAILING in 8 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242101/