[01:01:40] 10Release-Engineering-Team, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Andrew) [01:01:51] Documented ^ [04:12:21] PROBLEM - Free space - all mounts on deployment-mx02 is CRITICAL: (Service Check Timed Out) [04:17:14] RECOVERY - Free space - all mounts on deployment-mx02 is OK: OK: All targets OK [04:48:47] 10Release-Engineering-Team, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Andrew) I've looked at this for quite a while but I can't make any history version of "pytest modules/mtail/files" work locally no matter how far I go back in gi... [06:56:59] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:05] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93351 bytes in 0.929 second response time [07:36:56] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93661 bytes in 1.012 second response time [07:37:03] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52106 bytes in 0.981 second response time [08:10:23] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10akosiaris) >>! In T211881#6006347, @Milimetric wrote: > Recent events have compelled me to adopt grap... [08:11:08] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Regression, 10User-DannyS712: A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata - https://phabricator.wikimedia.org/T249018 (10Addshore) [08:13:40] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Regression, 10User-DannyS712: A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata - https://phabricator.wikimedia.org/T249018 (10DannyS712) [08:19:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Regression, 10User-DannyS712: A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata - https://phabricator.wikimedia.org/T249018 (10Addshore) So, diffs actually load fine that relate to the revision. but the revision... [08:32:03] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Regression, and 2 others: A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata - https://phabricator.wikimedia.org/T249018 (10Addshore) a:03Addshore [08:32:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Regression, 10User-Addshore, and 2 others: A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata - https://phabricator.wikimedia.org/T249018 (10Addshore) Moving to the iteration so I keep my eye on it and investigate... [09:47:40] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10akosiaris) A couple of things I... [09:52:53] 10Continuous-Integration-Config, 10VideoCutTool: Set up continuous integration for VideoCutTool - https://phabricator.wikimedia.org/T244079 (10Gopavasanth) 05Open→03Resolved [09:57:53] (03CR) 10Kosta Harlan: [C: 03+1] "PS4 looks good to me, thanks Brennen" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/584111 (https://phabricator.wikimedia.org/T246953) (owner: 10Kosta Harlan) [10:35:33] 10Gerrit, 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10jbond) 05Resolved→03Open I noticed that the disk on gerrit1002 was full again today, ` $ df -h Filesystem Size Used Avail Use% Mounted on udev 7.9G 0 7.9G 0% /dev tmpfs... [10:35:36] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10jbond) [11:37:35] (03Abandoned) 10Zfilipin: tutorial: Configure CI to run unit tests [blubber-doc/example/helloworldoid] - 10https://gerrit.wikimedia.org/r/575319 (owner: 10Zfilipin) [13:57:52] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations, 10serviceops: Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) [13:58:17] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10hashar) [13:58:19] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations, 10serviceops: Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) [14:09:46] (03PS12) 10Hashar: jjb: migrate refinery release job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) [14:10:43] (03CR) 10Hashar: "I have restored the email notification https://gerrit.wikimedia.org/r/#/c/integration/config/+/583092/11..12/jjb/analytics.yaml" [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [14:18:59] 10Release-Engineering-Team, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Joe) The problem is most probably we're installing a default mtail from buster, instead of the specific version we're using in production, which these tests rely on. [14:25:34] 10Release-Engineering-Team, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10hashar) CI has switched that job from Stretch to Buster with: 4e85d6332 zuul: [operations/puppet] Add buster as experimental 21b6dbfad zuul: [operations/puppet]... [14:26:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10hashar) [14:28:11] (03PS1) 10Hashar: Revert "jjb: Drop operations-puppet-tests-stretch-docker, no longer used" [integration/config] - 10https://gerrit.wikimedia.org/r/585239 (https://phabricator.wikimedia.org/T249076) [14:28:23] (03CR) 10Hashar: [C: 03+2] Revert "jjb: Drop operations-puppet-tests-stretch-docker, no longer used" [integration/config] - 10https://gerrit.wikimedia.org/r/585239 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [14:28:36] (03CR) 10Joal: "So cool :)" [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [14:29:15] (03Merged) 10jenkins-bot: Revert "jjb: Drop operations-puppet-tests-stretch-docker, no longer used" [integration/config] - 10https://gerrit.wikimedia.org/r/585239 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [14:32:08] (03PS1) 10Hashar: Revert "zuul: [operations/puppet] Use buster instead of stretch image" [integration/config] - 10https://gerrit.wikimedia.org/r/585241 (https://phabricator.wikimedia.org/T249076) [14:32:25] (03CR) 10Hashar: [C: 03+2] Revert "zuul: [operations/puppet] Use buster instead of stretch image" [integration/config] - 10https://gerrit.wikimedia.org/r/585241 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [14:32:30] (03CR) 10Joal: "Cool as well :) Thanks a lot Antoine for this work :)" [integration/config] - 10https://gerrit.wikimedia.org/r/584998 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [14:33:30] (03Merged) 10jenkins-bot: Revert "zuul: [operations/puppet] Use buster instead of stretch image" [integration/config] - 10https://gerrit.wikimedia.org/r/585241 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [14:43:00] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10hashar) p:05Triage... [14:44:01] James_F: good morning! I had to revert the puppet job upgrade from Stretch to Buster. Some test suite related on specific versions of Debian packages :/ [14:44:02] https://phabricator.wikimedia.org/T249076 [14:44:14] I guess those tests should be split to standalone jobs [14:44:32] hashar: Boo. [14:44:37] Yes. [14:44:44] Does anything pass now? [14:44:56] We had to upgrade to buster to let some changes pass. [14:45:01] They were merged. [14:45:14] So… the puppet repo is now broken? :-( [14:46:43] hashar: Or did you get SRE to revert that too? [14:46:51] oh nice [14:46:53] :-( [14:46:57] Yeah. [14:47:05] I just reverted a couple patches from integration/config [14:47:13] so I guess we need two jobs bah [14:47:17] That won’t fix it. It’ll just change the breakage. [14:47:21] Yeah. [14:47:24] :-\ [14:47:48] such that mtail test suite would be skip by default [14:47:56] and run on a Stretch container explicitly [14:48:07] while whatever other test was failling would be tied to Buster I guess [14:48:14] Yes. [14:48:22] as for the implementation, well I don't know [14:48:25] maybe that can be done via rake [14:48:31] like rake test [14:48:33] rake test:buster [14:48:35] rake test:stretch [14:48:38] something like that [14:48:57] anyway we can follow up on https://phabricator.wikimedia.org/T249076 an dgather the various breakage that occured previously [14:49:08] and start some inventory of which test suite requires which package/distro [14:49:18] I am out for an hour or so. Walking out with kids outside [14:50:28] ++ [14:58:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Andrew) Thanks! The... [15:02:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Jdforrester-WMF) p:... [15:04:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10Jdforrester-WMF) >>!... [15:04:43] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Consider what paging who should get for what in RelEng/EngProd? - https://phabricator.wikimedia.org/T247427 (10Jdforrester-WMF) [15:04:45] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10User-brennen: Investigate duration of scap sync for 1.35.0-wmf.23 train deploy - https://phabricator.wikimedia.org/T247426 (10Jdforrester-WMF) [15:04:47] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): check gerrit-replica healthcheck alerts - https://phabricator.wikimedia.org/T247186 (10Jdforrester-WMF) [15:04:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Fix or delete integration-config-qa - https://phabricator.wikimedia.org/T245037 (10Jdforrester-WMF) [15:04:51] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10Jdforrester-WMF) [15:04:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Replace integration-trigger-01 with running the dummy job on contint1001 agent - https://phabricator.wikimedia.org/T247109 (10Jdforrester-W... [15:04:55] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10CPT Initiatives (API Integration Tests), 10Code-Health, 10MW-1.35-notes (1.35.0-wmf.15; 2020-01-14): In CI, configure MediaWiki to enable API testing - https://phabricator.wikimedia.org/T243978 (10Jdforrester-WMF) [15:04:58] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Test upgrade of Gerrit to 2.16.13 on the ganeti test instance - https://phabricator.wikimedia.org/T243981 (10Jdforrester-WMF) [15:05:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10Jdforrester-WMF) [15:05:02] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Jdforrester-WMF) [15:05:07] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Test backup and restore of Gerrit on the ganeti test instance - https://phabricator.wikimedia.org/T243982 (10Jdforrester-WMF) [15:05:09] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Tune gerrit-replica to be more resilient to the kind of traffic it gets - https://phabricator.wikimedia.org/T243655 (10Jdforrester-WMF) [15:05:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: train-deploy-notes Jenkins job fails in conjunction with branch.py - https://phabricator.wikimedia.org/T243330 (10Jdforrester-WMF) [15:05:13] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Determine if it is possible to migrate remaining integration jessie permanent instances to stretch/buster - https://phabricator.wikimedia.org/T242909 (10Jdforrester-WMF) [15:05:16] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10CirrusSearch, and 4 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Jdforrester-WMF) [15:05:20] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [15:05:23] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) [15:05:37] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Developer Productivity, and 2 others: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Jdforrester-WMF) [15:05:37] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Research available third-party Argo and Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10Jdforrester-WMF) [15:05:40] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10wikimedia.biterg.io: Investigate wikimedia.biterg.io as possible replacement for people.wikimedia.org/~dduvall/jenkins - https://phabricator.wikimedia.org/T248351 (10Jdforrester-WMF) [15:05:42] 10Deployments, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-MModell: Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488 (10Jdforrester-WMF) [15:06:09] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-MModell, 10User-brennen: Make search context highlights work with the ferret search engine - https://phabricator.wikimedia.org/T230787 (10Jdforrester-WMF) [15:15:44] 10Continuous-Integration-Config, 10JsonConfig, 10Product-Infrastructure-Team-Backlog, 10Maps (Kartographer), and 2 others: Move JCMapDataContent from JsonConfig to Kartographer - https://phabricator.wikimedia.org/T163274 (10MSantos) 05Stalled→03Open [15:23:05] re [15:23:12] Hey. [15:29:08] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10thcipriani) 05Open→03Resolved... [15:30:17] thcipriani: Great! [15:30:51] James_F: unsatisfying, but glad it's not happening anymore (knock on wood) [15:31:26] Yes. [15:31:48] * James_F sighs. [15:31:54] Did I ever mention how much I hate Rake? [15:37:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10hashar) >>! In T249076#6019052, @Jdforrest... [15:38:19] hashar: Yes, but the puppet masters are running on buster. [15:38:32] yeah that is a point [15:38:33] We should align things in CI as much as possible to reality they're testing for, not the other way around. [15:38:41] then really mostly we care about the puppet version being used I guess [15:38:50] Ideally we'd have upgraded CI first BEFORE upgrading production. [15:38:54] * James_F coughs pointedly. [15:38:57] yeah [15:39:04] that is done by bumping the puppet version in the Gemfile [15:39:32] `gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 5.5.10'` [15:40:11] which is frozen when the container is build [15:40:14] and never updated after [15:40:16] * James_F nods. [15:40:16] :/ [15:40:50] We can trivially rev operations-puppet 0.6.2 and 0.7.0 to 0.6.3 and 0.7.1 if we need to. [15:54:39] (03PS1) 10Hashar: docker: operations-puppet back to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) [15:55:04] James_F: ^ that one just roll back to Stretch, though it still has the systemd package which would match the requirement from https://phabricator.wikimedia.org/rOPUPd70a7f24f2626f15765b376c738c6674e593e529 [15:55:11] and we can look at migrating to Buster after [15:56:24] Does it have the right pacakges for it? [15:56:46] Right versions/etc. [15:57:03] I'd like for _joe_ to weigh in, given we're reverting part of his work. [15:57:07] yeah [15:58:08] <_joe_> what's the issue? [15:58:17] (03CR) 10Hashar: "The mtail suite fails on Buster since it is geared at the package shipped by Stretch." [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [15:58:30] _joe_: The migration to buster for puppet CI broke mtail. [15:58:47] Want to check that you're OK with us rolling back to stretch with some patches to make it work. [15:59:03] <_joe_> James_F: I'm aware, and no [15:59:16] <_joe_> this didn't break *all* tox jobs [15:59:26] <_joe_> if it did, we have a regression in our code [15:59:29] Yeah. [15:59:40] <_joe_> but anyways, I'm working on a "proper" solution [15:59:50] OK. Should we hold off? [16:00:12] <_joe_> let me see what your patches did [16:00:30] I merely moved from ci-buster (which breaks the 'mtail' testsuite) to ci-stretch [16:00:44] Which breaks other things instead. [16:00:51] but I have kept the bit that drop tox as the parent image [16:00:56] Yeah. [16:00:57] and kept the addition of "systemd" [16:01:40] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "See my comment - I'd use 0.6.3 so I can roll out 0.7.1 tomorrow with the correct fix for buster." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [16:02:09] nitpick! :D [16:02:09] <_joe_> hashar: yeah see my comment, that is the right approach if we want to unbreak it now [16:02:29] <_joe_> well there is another one ofc [16:02:40] <_joe_> to just copy the right mtail binary :P [16:02:46] <_joe_> it's a static go binary anyways [16:02:53] That might "work". [16:03:06] But is there a tail (ho ho) of similar things we'll need to fix [16:03:07] ? [16:03:09] <_joe_> as an hotfix, I was going with that, but going back to stretch for a night is ok too [16:03:10] hmm [16:03:24] <_joe_> I prefer to work on a long-term solution [16:03:54] <_joe_> which includes probably paying back the tech debt of how the mtail jobs are tested :) [16:04:20] * James_F nods. [16:04:21] we could have a dedicated job just for mtail [16:04:27] <_joe_> but we might still need to ship multiple versions of mtail [16:04:33] <_joe_> hashar: well no the problem is deeper [16:04:47] <_joe_> we rely on an outdated version of mtail to run all tests [16:05:09] Is moving simple? [16:05:12] <_joe_> with a feature that has been removed from later versions [16:05:18] Ah. [16:05:30] <_joe_> James_F: I need to rewrite how they happen [16:05:48] <_joe_> but yes, it's not "hard" per se as far as CI is concerned [16:05:58] * James_F nods. [16:06:22] <_joe_> anyways, yes, let's rollback to stretch (which is a pity given the puppetmasters are on buster) [16:06:29] <_joe_> ahhh dang, no it would not work [16:06:33] <_joe_> sorry I just realized [16:06:37] then well we use the puppet version from Gemfile [16:06:38] (03PS2) 10Hashar: docker: operations-puppet back to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) [16:06:50] the puppet version being defined when we build the container via Gemfile.lock [16:06:51] <_joe_> stretch doesn't have systemd-analyze calendar [16:06:55] which is probably yet another bug hehe [16:07:04] <_joe_> no it's not [16:07:18] <_joe_> hashar: let's go with the other quick and dirty solution heh [16:07:28] Copying old mtail? [16:07:32] <_joe_> lemme get to my desk and I'll prepare it [16:07:35] <_joe_> James_F: 99% yes [16:07:39] Cool. [16:07:45] <_joe_> it's the quickest way to solve everything [16:07:56] and if systemd-analyze calendar is not available on Stretch, whenever https://phabricator.wikimedia.org/rOPUPd70a7f24f2626f15765b376c738c6674e593e529 is used on Stretch I guess we are in trouble aren't we? [16:07:58] <_joe_> hashar: sorry for the additional work on that patch [16:08:24] <_joe_> hashar: only if the puppetmaster is on stretch [16:08:28] ohh [16:08:35] <_joe_> generate() runs at compile time [16:08:43] yeah good point, I missed that [16:09:06] <_joe_> ok so, lemme go back and hotfix this somehow [16:09:33] https://gerrit.wikimedia.org/r/585261/2 should do [16:09:40] then I can bump the version in the jenkins job [16:10:23] <_joe_> hashar: no, because in this case, the puppetmaster it the docker image [16:11:09] <_joe_> catch-22 [16:13:26] eeek [16:13:34] then i guess I can craft a job dedicated to mtail [16:14:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel) [16:14:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel) [16:21:45] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10thcipriani) [16:21:45] <_joe_> ok so [16:21:59] <_joe_> we install 3.0.0~rc5-1~bpo9+1wmf2 on buster in prod [16:22:07] <_joe_> we have 3.0.0~rc19-2 in the docker image [16:22:12] <_joe_> because we do some apt pinning [16:22:25] <_joe_> I think I can find the solution [16:25:19] (03PS3) 10Hashar: docker: operations-puppet back to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) [16:25:21] (03PS1) 10Hashar: jjb: switch operations/puppet to 0.6.3 [integration/config] - 10https://gerrit.wikimedia.org/r/585267 (https://phabricator.wikimedia.org/T249076) [16:25:49] _joe_: or we can have a CI job based on Stretch that only runs tox -e mtail [16:26:03] and only triggers when files are changed under modules/mtail/ [16:26:07] that is reasonably easy to setup [16:26:32] but then we would need the rake magic module to no more run mtail [16:26:45] at least when the binary is not available [16:27:49] <_joe_> hashar: I'll have a fix in 10 minutes I think [16:42:46] (03PS1) 10Giuseppe Lavagetto: operations-puppet: pin a specific mtail version [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) [16:42:49] <_joe_> well more like 15 [16:43:34] <_joe_> I'm building it now to confirm [16:46:11] <_joe_> ok it builds and has the correct mtail version [16:46:46] <_joe_> hashar, James_F do you need me to do something else, or are you taking care of the deployment? [16:47:18] I had a cuple patches to revert to Stretch [16:47:47] or are you getting mtail in the Buster image as well? [16:48:11] <_joe_> hashar: see https://gerrit.wikimedia.org/r/585270 I just pinned the version [16:48:20] <_joe_> this fixes ci while I can work on a decent solution [16:49:14] _joe_: can we use something like apt install mtail/buster-wikimedia/thirdparty ? [16:49:18] instead of the explicit version [16:49:21] That plus updating the pointer in the jjb should suffice. [16:49:26] <_joe_> hashar: not really now, no [16:49:28] that just a rethorical question [16:49:28] ;D [16:49:41] <_joe_> it's a wrong solution too, we need to fix the apt config of that base image [16:49:46] Right. [16:49:58] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "Verified that this unbreaks tox" [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) (owner: 10Giuseppe Lavagetto) [16:50:04] okk ;) [16:50:11] <_joe_> James_F: I'm going afk, but I can be back later if needed [16:50:39] (03PS2) 10Hashar: operations-puppet: pin a specific mtail version [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) (owner: 10Giuseppe Lavagetto) [16:50:40] <_joe_> to be clearer, we might find many other such cases when moving to buster as base image [16:50:44] Yeah. [16:50:47] https://gerrit.wikimedia.org/r/#/c/integration/config/+/585270/1..2 [16:50:54] I have edited the changelog s/unreleased/wikidmedia [16:51:01] hashar: LGTM. Let's land it? [16:51:07] <_joe_> so that's sort of an UBN for my team, we'll work on it tomorrow [16:51:13] (03CR) 10Jforrester: [C: 03+1] operations-puppet: pin a specific mtail version [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) (owner: 10Giuseppe Lavagetto) [16:51:40] (03Abandoned) 10Hashar: docker: operations-puppet back to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/585261 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [16:52:07] (03Abandoned) 10Hashar: jjb: switch operations/puppet to 0.6.3 [integration/config] - 10https://gerrit.wikimedia.org/r/585267 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [16:52:12] James_F: yes ;) [16:52:18] (03CR) 10Hashar: [C: 03+2] operations-puppet: pin a specific mtail version [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) (owner: 10Giuseppe Lavagetto) [16:52:49] (03CR) 10Thcipriani: [C: 03+2] Add detailed release procedure documentation [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [16:53:21] (03Merged) 10jenkins-bot: operations-puppet: pin a specific mtail version [integration/config] - 10https://gerrit.wikimedia.org/r/585270 (https://phabricator.wikimedia.org/T249076) (owner: 10Giuseppe Lavagetto) [16:53:39] _joe_: thank you :] [16:54:36] (03Merged) 10jenkins-bot: Add detailed release procedure documentation [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [16:57:59] (03PS1) 10Hashar: jjb: update operations-puppet for mtail [integration/config] - 10https://gerrit.wikimedia.org/r/585274 (https://phabricator.wikimedia.org/T249076) [16:59:30] (03PS1) 10Hashar: Switch operations/puppet to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/585275 (https://phabricator.wikimedia.org/T249076) [17:00:22] docker-registry.discovery.wmnet/releng/operations-puppet:0.7.1 is building [17:01:36] hashar: There's a whole !log system for that. ;_p [17:01:40] hashar: Also, meeting. [17:04:37] (03CR) 10Hashar: [C: 03+2] "Updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/585274 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [17:05:04] (03CR) 10Hashar: [C: 03+2] "Should be good now." [integration/config] - 10https://gerrit.wikimedia.org/r/585275 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [17:05:40] (03Merged) 10jenkins-bot: jjb: update operations-puppet for mtail [integration/config] - 10https://gerrit.wikimedia.org/r/585274 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [17:06:04] (03Merged) 10jenkins-bot: Switch operations/puppet to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/585275 (https://phabricator.wikimedia.org/T249076) (owner: 10Hashar) [17:07:33] !log Reloading Zuul for https://gerrit.wikimedia.org/r/585275 # T249076 [17:07:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:07:35] T249076: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 [17:07:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error: Jenkins rejects all patches to operations/puppet/tox.ini - https://phabricator.wikimedia.org/T249076 (10hashar) 05Open→03... [17:08:17] _joe_: James_F: should be good now. The mtail fix is being tested on the dummy change https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/585058/ [17:08:24] 10Release-Engineering-Team-TODO, 10Scap, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10mmodell) this looks like it's likely the culprit: {rMSCA3ca8820b7a68b3f810cb3e975342c914e322b81c} [17:12:03] 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10mmodell) [17:12:21] 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10mmodell) p:05Triage→03High [17:14:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10ContentSecurityPolicy, and 3 others: .mp4 build artifacts not viewable due to CSP in Chrome - https://phabricator.wikimedia.org/T245658 (10thcipriani) [17:15:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10ContentSecurityPolicy, and 3 others: .mp4 build artifacts not viewable due to CSP in Chrome - https://phabricator.wikimedia.org/T245658 (10thcipriani) p:... [17:17:08] 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10thcipriani) >>! In T246793#6019857, @mmodell wrote: > this looks like it's likely the culprit: {rMSCA3ca8820b7a68b3f810cb3e975342c914e322b8... [17:19:10] 10Release-Engineering-Team-TODO, 10Operations: Should 'doc' machines (i.e. doc1001) have contint-roots as a group? - https://phabricator.wikimedia.org/T245691 (10thcipriani) a:03thcipriani [17:20:01] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: Dockerize our Phabricator development environment - https://phabricator.wikimedia.org/T245575 (10thcipriani) [17:20:15] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: Dockerize our Phabricator development environment - https://phabricator.wikimedia.org/T245575 (10thcipriani) p:05Triage→03Medium [17:20:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10thcipriani) p:05Triage→03Medium [17:22:15] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release Pipeline: PipelineLib needs a timeout - https://phabricator.wikimedia.org/T244892 (10thcipriani) [17:22:38] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release Pipeline: PipelineLib needs a timeout - https://phabricator.wikimedia.org/T244892 (10thcipriani) p:05Triage→03Medium [17:25:59] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Developer Productivity, and 2 others: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10thcipriani) a:05thcipriani→03None [17:30:17] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Build a Seakeeper MVP - https://phabricator.wikimedia.org/T248876 (10LarsWirzenius) p:05Medium→03High [17:30:29] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Research available third-party Argo and Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10dduvall) p:05High→03Medium [17:33:13] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Consider what paging who should get for what in RelEng/EngProd? - https://phabricator.wikimedia.org/T247427 (10mmodell) Note that I get paged by phab emails by way of an `email`->`push notification` gateway + app on my phone. [17:37:12] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Documentation, 10Kubernetes, 10Patch-For-Review: Document how to migrate a service to kubernetes - https://phabricator.wikimedia.org/T248916 (10thcipriani) p:05Triage→03Medium [17:45:17] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar... [17:45:33] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release Pipeline (Blubber): Blubber policy should be verified after expansion - https://phabricator.wikimedia.org/T248927 (10thcipriani) p:05Triage→03Medium [17:47:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: add folders to jenkins artifacts - https://phabricator.wikimedia.org/T248373 (10hashar) a:05hashar→03None I am not working on this, but you can follow my... [17:47:31] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10wikimedia.biterg.io: Investigate wikimedia.biterg.io as possible replacement for people.wikimedia.org/~dduvall/jenkins - https://phabricator.wikimedia.org/T248351 (10thcipriani) p:05Triage→03Low [17:47:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10observability: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113 (10hashar) 05Open→03Resolved [17:52:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Patch-For-Review, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) [17:58:15] phab question: in my personal navbar i have a project procurement (when clicked goes to workboard) and project dc-ops (when clicked goes to manage project) [17:58:25] no clue why they differ in navigation as both are project links. [17:58:46] i want both to just go to the main landing or workgboard, never default to manage (that way lies misconfiguration) [18:00:03] robh: I'm guessing it's following the default for that project [18:00:05] navbar? the top bar? under bookmarks? [18:00:08] robh: there is a 'manage menu' option under the 'manage project' view [18:00:10] yeah, probably that ^ [18:00:18] IIRC, some projects have default to workboard, some ot the project view [18:00:22] ok, i vaguely recall that but i couldnt find, checking [18:00:24] each project can select the default page [18:00:41] there is an edit menu [18:00:58] https://phabricator.wikimedia.org/project/1155/item/configure/global/ [18:01:13] click the thumbtack next to the page you want to make the default [18:01:20] yeah, done [18:01:21] thank you! [18:01:28] this was bugging me so much for like a month. [18:01:38] hrmm.. i didnt do it right [18:02:01] ahh got it nm. [18:02:13] thank you all =] [18:02:17] np [18:02:28] since we're directing everyone to the dc ops page, helps if it defaults to the proper thing [18:02:32] rather than trying to send everyone to edit it. [18:06:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10hashar) 05Open→03Declined The i... [18:06:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Test-Coverage: Switch mediawiki code coverage from xdebug to phpdbg or pcov - https://phabricator.wikimedia.org/T234020 (10hashar) [18:17:18] Ok,next question. on my dashboard https://phabricator.wikimedia.org/dashboard/view/47/ I have on-site queues panel. when i view this, no matter what it lists the ops-codfw tab of that panel by default, making me click to the other tabs, despite tab order [18:17:41] is it just alphabetical load by tab name no matter what for a panel? [18:18:39] by project atleast, as i renamed it '1 ulsfo' and '2 codfw' and it didnt reorder. [18:19:21] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Daimona) 05Declined→03Open >>!... [18:19:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Test-Coverage: Switch mediawiki code coverage from xdebug to phpdbg or pcov - https://phabricator.wikimedia.org/T234020 (10Daimona) [18:47:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:51] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93351 bytes in 0.937 second response time [19:14:27] (03CR) 10Kosta Harlan: dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) (owner: 10Daimona Eaytoy) [19:23:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:50] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93350 bytes in 1.751 second response time [19:34:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: add folders to jenkins artifacts - https://phabricator.wikimedia.org/T248373 (10zeljkofilipin) a:03zeljkofilipin [19:42:44] robh: looks like there is no way to change the default tab. tabbed panels are a little bit buggy it seems [19:42:48] cam [19:42:59] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T247773 (10dduvall) [19:43:01] I can't seem to reorder by moving the first tab either [19:43:02] yeah, oh well i removed the tabbed panel and ust put in a single pane [19:54:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:31] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10DannyS712) [20:22:48] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10DannyS712) p:05Triage→03Low [20:23:27] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10DannyS712) Although per https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_Projects this could be created without prior discussion, filing this task to see if there are any o... [20:41:50] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10Ammarpad) [21:14:31] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar... [21:40:35] !log gerrit: deleting old sandbox branches in mediawiki/core that were used for testing purpose # T227159 [21:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:40:37] T227159: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 [21:40:42] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907): Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10hashar) We still had the sandbox branches left over: origin/sandbox/hashar/hexmode 38d4ec5d81 origin/sandb... [21:41:13] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10Aklapper) Are stakeholders / hook code stewards fine with this and do they plan to use this project tag? [22:07:03] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10Krinkle) FYI: Components named in this way have often become the destination for any task relating to a particular system. For example, I would not be surprised if a problem with using CentralAu... [22:18:43] (03CR) 10Daimona Eaytoy: dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) (owner: 10Daimona Eaytoy) [22:32:04] (03PS7) 10Daimona Eaytoy: dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+ [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) [22:38:53] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:49:53] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:28:30] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Hooks - https://phabricator.wikimedia.org/T249170 (10DannyS712) There are no listed maintainers for the hooks (so no clear code stewards) Makes sense regarding names, slight preference for `MediaWiki-Core-Hooks` so that its clear it also affects o... [23:39:35] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Core-Hooks - https://phabricator.wikimedia.org/T249170 (10DannyS712)