[00:25:34] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:34:21] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [00:45:34] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:52:23] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:06:35] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:11:35] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:16:59] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:18:29] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [08:46:53] hashar: bonjour, could we give https://gerrit.wikimedia.org/r/c/integration/config/+/983476/ a try? [09:36:25] 10Release-Engineering-Team (Radar), 10Anti-Harassment, 10Security-Team, 10iPoid-Service, 10Patch-For-Review: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10kostajh) >>! In T338238#9403317, @mmartorana wrote: > Hi @kostajh - I have created [[ https://gitlab.wikimedia... [10:31:37] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10Aklapper) [10:32:04] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10Aklapper) I uninstalled Differential. [11:45:00] 10Continuous-Integration-Config, 10MediaWiki-extensions-Interwiki, 10MediaWiki-extensions-Translate, 10TranslationNotifications: Add i18n related extensions to the CI gate - https://phabricator.wikimedia.org/T86930 (10Nikerabbit) [12:25:10] Which of my dear releng coworkers discussed Phab MariaDB access and checking config.yaml with me in person 10 days ago? (Sorry, can't remember) [12:31:44] 10Phabricator, 10Wikimedia-Phabricator-Extensions: Remove custom src/customfields/DifferentialApplyPatchWithOnlyGitField.php - https://phabricator.wikimedia.org/T342041 (10Aklapper) [13:36:57] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Refactor golang-gosec CI include - https://phabricator.wikimedia.org/T353534 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates/-/merge_r... [15:25:48] 10GitLab (Project Migration), 10Data-Platform-SRE: Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10Gehel) [15:31:20] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Reedy) [15:54:34] andre: that was me I think :) [15:56:58] I am in vacations though :] [15:57:27] kostajh: I am in vacations this week, so I am afraid that will have to wait a couple weeks [15:57:59] (I changed my nickname to make it clear) [16:05:06] Have a great Christmas [16:25:45] 10GitLab, 10collaboration-services: Install Gitlab slack app - https://phabricator.wikimedia.org/T353034 (10LSobanski) Read-only (e.g. notifications) is straightforward, write operations (slash commands) would require a security review. [16:31:17] 10GitLab, 10collaboration-services: Review the capabilities of the Gitlab slack app - https://phabricator.wikimedia.org/T353034 (10LSobanski) [16:31:52] 10GitLab, 10collaboration-services: Review the capabilities of the Gitlab slack app - https://phabricator.wikimedia.org/T353034 (10LSobanski) p:05Triage→03Low [17:05:07] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10Dzahn) woohoo, that's kind of a milestone. Thanks! [17:08:53] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [17:09:08] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Refactor golang-gosec CI include - https://phabricator.wikimedia.org/T353534 (10sbassett) 05Open→03Resolved p:05Triage→03Medium [17:13:09] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Create rule-filtering functionality for semgrep-merge-tool rule repos - https://phabricator.wikimedia.org/T353536 (10sbassett) p:05Triage→03Medium [17:13:17] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10collaboration-services, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) @Aklapper Yea, weak "yes". The cloud one, sure, just a comment and not relevant anymore. The other... [17:17:46] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10collaboration-services, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) it never ends: ` data/fixed_settings.yaml:diffusion.ssh-user: 'vcs' files/phab_deploy_config_dep... [17:26:57] (03PS1) 10Ahmon Dancy: Add rsync to node*-test images. [integration/config] - 10https://gerrit.wikimedia.org/r/983892 [17:42:49] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Refactor golang-gosec CI include - https://phabricator.wikimedia.org/T353534 (10sbassett) [17:55:54] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Refactor golang-gosec CI include - https://phabricator.wikimedia.org/T353534 (10sbassett) Alright, great - it looks like the changes are working as expected, from the output of a job in our CI test... [17:58:00] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10phan-taint-check-plugin, 10SecTeam-Processed, and 2 others: phan-taint-check gitlab ci security template should use its own image - https://phabricator.wikimedia.org/T352820 (10sbassett) [19:51:20] 10GitLab-Test, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10dancy) [19:52:11] 10GitLab-Test, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10dancy) @Jelto @dduvall FYI [20:30:36] 10Release-Engineering-Team, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10XenoRyet) [20:52:45] (03PS1) 10BCornwall: Archive operations/debs/dnsdist [integration/config] - 10https://gerrit.wikimedia.org/r/983940 (https://phabricator.wikimedia.org/T347623) [21:11:06] 10GitLab-Test, 10Release-Engineering-Team, 10cloud-services-team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10bd808) p:05Triage→03Medium +1 [21:17:09] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [21:44:55] 10GitLab-Test, 10Release-Engineering-Team, 10cloud-services-team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10Andrew) a:03Andrew [21:55:23] 10GitLab-Test, 10Release-Engineering-Team, 10cloud-services-team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10Andrew) 05Open→03Resolved I couldn't make the cookbook cooperate but I adjusted the quotas vi the openstack cli. [22:17:41] 10GitLab-Test, 10Release-Engineering-Team, 10cloud-services-team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10Andrew) I also boosted your instance count by 2 in case you need space to shuffle things. Hopefully in-place resiz... [22:18:33] 10GitLab-Test, 10Release-Engineering-Team, 10cloud-services-team, 10Cloud-VPS (Quota-requests): Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10dancy) Thanks a lot! I just tried an in-place resize for zuul-1001 and it worked great! [22:25:34] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/33 Remove unused "Differential" application from default settings [22:25:36] 10Phabricator (phabricator-next), 10Voice & Tone: When a user clicks on Create Project but doesn't have permission to create a project, provide a link to learn how to create a project - https://phabricator.wikimedia.org/T165192 (10CodeReviewBot) aklapper closed https://gitlab.wikimedia.org/repos/phabricator/de... [22:25:46] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10CodeReviewBot) aklapper closed https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/28 Remove unused "Differential" application from default settings [22:40:52] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10CodeReviewBot) aklapper merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/33 Remove unused "Differential" application from default settings [22:41:50] 10Phabricator (phabricator-next): Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10Aklapper) [23:18:49] 10Release-Engineering-Team, 10Scap, 10SRE: Php-fpm failed to start on one host during Scap - https://phabricator.wikimedia.org/T353679 (10Mstyles) [23:30:39] 10Release-Engineering-Team, 10Scap, 10SRE: Php-fpm failed to start on one host during Scap (ssh to mw2448.codfw.wmnet timed out) - https://phabricator.wikimedia.org/T353679 (10bd808) [23:31:46] 10Release-Engineering-Team, 10Scap, 10SRE: Php-fpm failed to start on one host during Scap (ssh to mw2448.codfw.wmnet timed out) - https://phabricator.wikimedia.org/T353679 (10dancy) Thanks for filing this! There have been several reports of mw2448 going offline this year: T345597, T334429 + T334420 (probab... [23:39:18] 10Release-Engineering-Team, 10Scap, 10SRE: Php-fpm failed to start on one host during Scap (ssh to mw2448.codfw.wmnet timed out) - https://phabricator.wikimedia.org/T353679 (10taavi) ` ------------------------------------------------------------------------------- Record: 3 Date/Time: 12/18/2023 19:48... [23:52:18] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10collaboration-services, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Aklapper) @Dzahn: Eh, sorry if I opened a can of worms. Feel free to leave as-is then?