[01:21:24] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12387904 (10Lars) Hi folks (@Peter @hashar @Andrew ), our CI tim... [04:39:10] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12388093 (10Peter) Hi @Lars I checked the job in CI and I think... [04:49:04] (03PS1) 10Phedenskog: civicrm: keep the MySQL data of the CiviCRM job in memory [integration/config] - 10https://gerrit.wikimedia.org/r/1348150 (https://phabricator.wikimedia.org/T434024) [04:50:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12388096 (10Peter) A made patch, please have a look that it looks ok. With the databa... [07:43:34] (03PS1) 10Hashar: jjb: pass FORCE_COLOR/TERM in fresh-test VM [integration/config] - 10https://gerrit.wikimedia.org/r/1348401 [07:48:28] (03CR) 10Hashar: [C:03+2] jjb: pass FORCE_COLOR/TERM in fresh-test VM [integration/config] - 10https://gerrit.wikimedia.org/r/1348401 (owner: 10Hashar) [07:50:08] (03Merged) 10jenkins-bot: jjb: pass FORCE_COLOR/TERM in fresh-test VM [integration/config] - 10https://gerrit.wikimedia.org/r/1348401 (owner: 10Hashar) [08:33:24] (03PS6) 10Hashar: test: colorize output of assertions [fresh] - 10https://gerrit.wikimedia.org/r/1347656 [09:24:36] Project mediawiki-core-doxygen build #23260: 04FAILURE in 6 min 33 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23260/ [09:54:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 07ContentSecurityPolicy: Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735#12388521 (10Osps7) I’d like to work on this task. From the current investigation, the issue appears t... [10:06:52] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Epic, and 2 others: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#12388576 (10MLechvien-WMF) [10:10:03] maintenance-disconnect-full-disks build 847140 integration-agent-docker-1092 (/: 29%, /srv: 95%, /var/lib/docker: 35%): OFFLINE due to disk space [10:15:03] maintenance-disconnect-full-disks build 847141 integration-agent-docker-1092 (/: 29%, /srv: 75%, /var/lib/docker: 34%): RECOVERY disk space OK [11:55:20] (03update) 10jelto: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) [12:04:21] (03open) 10jelto: digitaocean: use new vpc egress gateway and isolated_workers in production [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/653 (https://phabricator.wikimedia.org/T425441) [12:32:24] Yippee, build fixed! [12:32:24] Project mediawiki-core-doxygen build #23261: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23261/ [12:38:54] (03CR) 10Hashar: [C:03+2] "Nice one! I have updated the job and retriggered a few builds (eg https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-b" [integration/config] - 10https://gerrit.wikimedia.org/r/1348150 (https://phabricator.wikimedia.org/T434024) (owner: 10Phedenskog) [12:40:46] (03Merged) 10jenkins-bot: civicrm: keep the MySQL data of the CiviCRM job in memory [integration/config] - 10https://gerrit.wikimedia.org/r/1348150 (https://phabricator.wikimedia.org/T434024) (owner: 10Phedenskog) [12:55:17] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12389127 (10WMDE-leszek) Hello, I kindly request that my two WMDE colleagues @SuzanneWood-WMDE and @seanleong-WMDE are added to acl*Project-admins. They're reg... [14:10:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12389445 (10hashar) I have manually rebuild the job a 20 times w... [14:12:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12389448 (10Peter) Also FYI T439729 upcoming changes for which g... [14:36:05] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:38:15] ^ that deployment-puppetserver alert has been going on for a while [14:48:18] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-CodeMirror] Add CodeMirror job [integration/config] - 10https://gerrit.wikimedia.org/r/1347709 (https://phabricator.wikimedia.org/T433073) [14:49:05] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:51:05] FIRING: InstanceDown: Project deployment-prep instance deployment-eventstreams-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:51:14] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-eventstreams-2 is down - https://phabricator.wikimedia.org/T439989 (10wmcs-alerts) 03NEW [14:52:26] (03CR) 10Vaughn Walters: [C:03+2] "merging - tests pass on https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-CodeMirror/1/console" [integration/config] - 10https://gerrit.wikimedia.org/r/1347709 (https://phabricator.wikimedia.org/T433073) (owner: 10Vaughn Walters) [14:52:34] taavi: I'll take a look [14:54:34] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-CodeMirror] Add CodeMirror job [integration/config] - 10https://gerrit.wikimedia.org/r/1347709 (https://phabricator.wikimedia.org/T433073) (owner: 10Vaughn Walters) [14:55:54] taavi: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1058199 has conflicts after `git pull --rebase` [14:56:05] RESOLVED: [2x] InstanceDown: Project deployment-prep instance deployment-eventgate-4 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:04:32] taavi: Correction. It's https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013 [15:24:18] taavi: Resolved. [15:31:05] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:32:02] neat, thanks. [15:45:56] 06Release-Engineering-Team (Priority Backlog šŸ“„), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219#12390152 (10Sgs) ##### Risky Patch! šŸš‚šŸ”„ * **Change**: [[ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CommunityConfigur... [16:03:00] !log krinkle@doc2003:/srv/doc/cover$ sudo -u doc-uploader rm -rf mediawiki-libs-node-cssjanus/; krinkle@doc1004:/srv/doc/cover$ sudo -u doc-uploader rm -rf mediawiki-libs-node-cssjanus/; ref T424419 [16:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:03:03] T424419: cssjanus.js has two coverage reports on https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T424419 [16:03:36] 10Continuous-Integration-Config, 10CSSJanus, 06MediaWiki-Core-Platform-Team (Kanban), 07Test-Coverage: cssjanus.js has two coverage reports on https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T424419#12390255 (10Krinkle) 05Open→03Resolved [16:12:36] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12390305 (10brennen) > I kindly request that my two WMDE colleagues @SuzanneWood-WMDE and @seanleong-WMDE are added to acl*Project-admins. Done! //Standard... [16:32:03] FIRING: InstanceDown: Project deployment-prep instance deployment-schema-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:32:09] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-schema-3 is down - https://phabricator.wikimedia.org/T440012 (10wmcs-alerts) 03NEW [16:37:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-schema-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:50:58] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Epic, and 2 others: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#12390860 (10Scott_French) Thanks for all of your work here, @dancy! As expected, the test deployment of shellbox yesterday went smoothly. The only question I ha... [19:42:40] 06Release-Engineering-Team (Priority Backlog šŸ“„), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218#12391004 (10brennen) 05Open→03Resolved Train seems stable on all wikis, although I'm not thrilled with the current state o... [20:05:20] (03open) 10dancy: deploy-service: Announce each group of the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1340 (https://phabricator.wikimedia.org/T412941) [20:06:29] (03update) 10dancy: deploy-service: Announce each group of the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1340 (https://phabricator.wikimedia.org/T412941) [20:35:28] (03PS3) 10Thcipriani: wmf-deployment: remove create/push for wmf/* branches [mediawiki] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1346021 (https://phabricator.wikimedia.org/T439548) [20:36:20] (03CR) 10Thcipriani: "Done" [mediawiki] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1346021 (https://phabricator.wikimedia.org/T439548) (owner: 10Thcipriani) [20:43:15] (03approved) 10swfrench: deploy-service: Announce each group of the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1340 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [21:43:37] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM!" [mediawiki] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1346021 (https://phabricator.wikimedia.org/T439548) (owner: 10Thcipriani) [21:45:47] (03update) 10dancy: deploy-service: Announce each group of the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1340 (https://phabricator.wikimedia.org/T412941) [21:45:59] (03merge) 10dancy: deploy-service: Announce each group of the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1340 (https://phabricator.wikimedia.org/T412941) [21:46:45] (03open) 10dancy: Release 4.294.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1341 [21:46:46] (03update) 10dancy: Release 4.294.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1341 [21:49:27] (03merge) 10dancy: Release 4.294.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1341 [23:47:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12391871 (10Dzahn) 05Open→03Resolved This should be resolved now! The ferm rules for 2181/2182/2183 were correct (see the counte...