[00:06:41] https://gerrit-review.googlesource.com/c/homepage/+/225992 gerrit community managers! [01:32:41] (03PS1) 10Legoktm: Configure tox for labs/tools/shorturls [integration/config] - 10https://gerrit.wikimedia.org/r/512794 [01:33:32] (03CR) 10Legoktm: [C: 03+2] Configure tox for labs/tools/shorturls [integration/config] - 10https://gerrit.wikimedia.org/r/512794 (owner: 10Legoktm) [01:35:03] (03Merged) 10jenkins-bot: Configure tox for labs/tools/shorturls [integration/config] - 10https://gerrit.wikimedia.org/r/512794 (owner: 10Legoktm) [01:35:30] !log deployed https://gerrit.wikimedia.org/r/512794 [01:35:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:51:04] 10Project-Admins, 10Phlogiston: Rename #Category to #Category-Phlogiston - https://phabricator.wikimedia.org/T224450 (10Aklapper) [04:16:54] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [07:01:54] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:04:11] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:14:01] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:48:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) Cutting the branch: https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_G... [08:10:31] Gerrit is not working properly for me. For example the dashboad doesn't work at all [08:11:16] it is ok for me as a logged in user [08:12:18] Seems to work in icognito mode... wth is my browser doing then [08:13:28] restarted browser, dashboard still not loading [08:15:30] another browser? [08:15:33] out of curiosity [08:21:34] seems completely broken now [08:22:09] people are looking at it in another channel [08:31:56] Project beta-code-update-eqiad build #248509: 04FAILURE in 8 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/248509/ [08:33:51] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Volans) Restarted gerrit because was stuck and showed the same behaviour of the above graph: https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId... [08:34:19] Yippee, build fixed! [08:34:20] Project beta-code-update-eqiad build #248510: 09FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/248510/ [08:35:48] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 5.56% of data above the critical threshold [3.0] [08:38:37] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [08:39:13] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Volans) As suggested by @dcausse let's try to capture a jstack next time it happens: ` sudo -u gerrit2 jstack $(pidof java) ` [08:40:28] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) [08:40:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) [08:40:46] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) [08:41:02] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) p:05Triage→03Unbreak! [08:42:00] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) @hashar @thcipriani any idea what went wrong? [08:43:59] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:19:05] zeljkof: moare or lessa rrived at home :) [09:20:06] hashar: great :) let me know when you have a few minutes to help me with https://phabricator.wikimedia.org/T224453 [09:21:11] zeljkof: have you looked at the logs in logstash ? [09:21:29] hashar: no, not sure which logs to look at [09:21:31] logstash.wikimedia.org , then at the top there is a link to a "scap" dashboar [09:21:31] d [09:21:48] ah, there's scap dashboard? :) didn't know about that, looking [09:21:52] scap sends its logs with type=scap [09:22:07] the thing is [09:22:19] that scap handling of subcommands is fundamentally broken in multiple various way [09:22:35] but you might try scap --verbose prep [09:22:40] which might dump the full logs to the console [09:22:57] else, since the trace shows it is breaking when doing the git submodule update, just try again [09:22:58] anywy [09:23:16] the root cause is 08:29 restarting gerrit due to stack threads - T224448 [09:23:17] T224448: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 [09:23:21] (gerrit was being restarted). [09:23:41] ah, that was all? [09:24:04] can I just retry the command? or do I need to do some cleanup first? [09:24:41] your guess? [09:25:17] to rephrase what I said: the git submodule update for extensions failed. [09:25:47] so it might have initialized/updated some submodules, but rerunning the git submodule update for them is not going to change anything so it is fine [09:25:58] gerrit was down, so at some point submodule failed [09:27:32] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10hashar) [09:27:34] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10hashar) 05Open→03Resolved a:03hashar From the stacktrace, `git submodule update` failed. Due to Gerrit being restarted: ` 08:29 restarting gerrit due to stack threads - T224448 ` [09:37:46] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) I've tried it again: ` zfilipin@deploy1001:/srv/mediawiki-staging$ scap prep 1.34.0-wmf.7 09:36:33 Version already checked out ` [09:37:54] hashar: ok, so I'm not sure this is good :/ https://phabricator.wikimedia.org/T224453#5215980 [09:38:08] I get `Version already checked out` [09:38:41] I'm not sure if I can trust it and continue? or should I delete what's there and try again? [09:46:08] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10hashar) We have plenty of stacktraces already and the one in this task description matches :-] The sendEmail thread is on hold waiting for some lock which is a... [09:50:50] hashar: quick q: is it possible to say to jenkins to execute the dpeloyment pipeline job that creates the docker images on git push rather than on gerrit merge? [09:51:11] zeljkof: just run git submodule update [09:51:56] mobrovac: no [09:52:17] mobrovac: well hmm . Yes it could but really a git push means bypassing everything isn't it ? ;d [09:53:27] hashar: i'm aware of that, yes, the reason is that i'm asking that for restbase; we are splitting in two services (api router and storage) and all of the dev and tests will still happen on gh, and we manually push to gerrit for deploys [09:53:48] hashar: the build pipeline will be used for the front-end, api routeing component only [09:53:59] and because they share code, we don't want them in two repos [09:55:13] to be checked with the pipeline team / wg , but I don't think you can continue working on github [09:55:28] or maybe and just then send a change for review in gerrit [09:55:37] which then enter the pipeline just like other services? [09:56:00] that won't work [09:56:07] cause really, if you do everything on github (test/dev etc) that is not what our pipeline is [09:56:26] the pipeline expcets a change in gerrit, then some kind of vote (cr+2) which then let the change enter the pipe [09:56:31] I would rather not support a straight git push [09:56:39] there is the same issue with ORES [09:57:01] which somehow get developed entirely on github then their team ask us to add patches and exception in to match their workflow [09:57:14] so really I don't know. That is for me out of the scope. [09:58:09] mobrovac: tldr, do check with the others from the ssd pipeline working group? But my stance is that it must be a change send for review and then receiving CR+2 [09:58:31] that being said, [09:58:40] we can have Zuul to react on a branch being updated (eg via a git push) [09:59:22] hm now that you've said "branch", we could have a different branch that we submit patches [09:59:41] but my main problem with this is that we really mirror from github into gerrit [09:59:43] just have to use the "post" pipeline which reacts whenever a branch is updated ( /refs/heads/.* ) [09:59:55] yes, that was my initial quesiton [10:00:13] I dereailled sorry :) [10:00:41] or really, just migrate to Gerrit/Phabricator. Solved! :] [10:09:28] hashar: just to make sure I don't mess up something :/ where should I run `git submodule update`? at `/srv/mediawiki-staging` or `php-1.34.0-wmf.7`? [10:09:46] `php-1.34.0-wmf.7`? [10:10:27] zeljkof: does that directory has any submodules ? [10:10:32] and if so, are they initialized? :) [10:10:41] (initialized/upto date) [10:11:08] both folders have submodules, as far as I can see [10:12:15] git submodule status [10:12:19] will tell you :) [10:12:21] then update [10:13:02] I'm just reading the docs, I've never worked much with submodules :/ [10:14:47] ok, so `php-1.34.0-wmf.7` folder has many submodules [10:15:11] `zfilipin@deploy1001:/srv/mediawiki-staging/php-1.34.0-wmf.7$ git submodule status` returns a lot of them with `(heads/master)` [10:15:25] and a few without that [10:15:32] so I guess I need to do this: [10:16:09] `zfilipin@deploy1001:/srv/mediawiki-staging/php-1.34.0-wmf.7$ git submodule update` [10:16:15] hashar: correct? ^ [10:16:33] * zeljkof doesn't want to break things [10:20:00] ok, done, no problems [10:20:23] :D [10:20:38] * zeljkof hates train [10:34:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) @herron sorry for the misleading flavor name. I indeed thought about using the 8vCPUs ones: `m1.xlar... [10:41:33] 10Gerrit: Delete "extensions/3D" repository? - https://phabricator.wikimedia.org/T224463 (10Aklapper) [10:58:36] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10zeljkofilipin) [10:58:48] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10zeljkofilipin) p:05Triage→03Unbreak! [11:00:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) [11:00:43] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10zeljkofilipin) [11:01:25] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10zeljkofilipin) Caused by T224453? [11:03:11] 10Scap: `scap prep` fails with `Unhandled error` - https://phabricator.wikimedia.org/T224453 (10zeljkofilipin) Tried to clean up things with this: ` zfilipin@deploy1001:/srv/mediawiki-staging/php-1.34.0-wmf.7$ git submodule update ` as suggested by @hashar in `#wikimedia-releng`. [11:13:06] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Paladox) This looks to be the same as what someone else had https://bugs.chromium.org/p/gerrit/issues/detail?id=7645 [11:22:51] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10hashar) >>! In T224465#5216144, @zeljkofilipin wrote: > Caused by T224453? That was `git submodule update` failing due to Gerrit being restarted. Your task is `rebuildLocalisationCache.php` failing after 3 minutes.... [11:31:37] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10hashar) >>! In T224033#5213715, @Joe wrote: > I am 100% against having ci handle merges of ops/puppet. Think of the case ci is down and we need puppet for anything. Seem my thi... [11:48:43] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, and 3 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Urbanecm) a:03Volans I was able to deploy two patches without any problems, so I guess ev... [11:50:37] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10User-Urbanecm, and 2 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Volans) 05Open→03Resolved Glad to hear, resolving then. [11:52:43] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Issue with inter-extension-dependencies in CI tests - https://phabricator.wikimedia.org/T224285 (10hashar) I am not sure that is the reason for this change, but the non BlueSpice repositories are using the `master` branch, **not the `REL... [11:55:22] 10Continuous-Integration-Config: Issue with inter-extension-dependencies in CI tests - https://phabricator.wikimedia.org/T224285 (10hashar) [11:55:46] zeljkof: have you got the branch cut / security patches applied ? [11:55:55] yes [11:56:16] got stuck at pushing to testwiki, waiting for swat to finish, so I'll continue [11:56:16] \o/ [11:56:26] ah yeah swat [11:56:33] well done [11:56:48] cause last tuesday and the tuesday before, I did the testwiki scap right during swat [11:56:58] which was hmm, annoying for the SWATers ;D [12:09:28] Lol [12:43:49] !log delete extensions/3D repo in gerrit - T224463 [12:43:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:43:52] T224463: Delete "extensions/3D" repository? - https://phabricator.wikimedia.org/T224463 [12:43:57] 10Gerrit: Delete "extensions/3D" repository? - https://phabricator.wikimedia.org/T224463 (10Paladox) 05Open→03Resolved a:03Paladox That is a mistake. I've deleted it now. [12:56:41] I think I'm making a really basic mistake here... I've added a new dependency in composer.json, https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/FileImporter/+/512675/ . I see the packages get installed in CI, yet autoloading fails to look up some new sub-dependencies. [12:57:02] Do I need to duplicate dependencies into dev-dependencies? [12:58:37] you might need load_composer_autoloader in extension.json: https://www.mediawiki.org/wiki/Manual:Extension.json/Schema#load_composer_autoloader [12:59:35] but also, the new dependency will have to go through security review and be added to mediawiki-vendor, if it’s not in there yet [12:59:42] https://www.mediawiki.org/wiki/Manual:External_libraries [13:00:39] before this can be deployed to production [13:00:52] Good point, thanks! [13:01:35] Also, load_composer_autoloader does look like the clue I needed :-) [13:06:41] Looks like that flag isn't enough to fix the phan job--but the security review is enough of a blocker that I think I'll revert the new library anyway. [13:15:35] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 4 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10Eevans) [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:29:40] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10dcausse) Here SendEmail-1 is not "blocked" it's just waiting for jobs, however the dump says: `Locked ownable synchronizers:- <0x00000001c13617f8> (a java.util.... [13:36:09] (03PS2) 10Kosta Harlan: Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 [13:48:27] (03CR) 10Kosta Harlan: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [13:56:11] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani) >>! In T224448#5216469, @dcausse wrote: > Here SendEmail-1 is not "blocked" it's just waiting for jobs, however the dump says: > `Locked ownable syn... [13:56:49] (03CR) 10Cparle: [C: 03+1] Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [14:13:46] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) [14:13:48] 10Scap: `scap sync` fails with CalledProcessError - https://phabricator.wikimedia.org/T224465 (10zeljkofilipin) 05Open→03Resolved a:03thcipriani Caused by T224453. Fixed by (thanks to @thcipriani ): ` cd /srv/mediawiki-staging/ rm -rf php-1.34.0-wmf.7 scap prep 1.34.0-wmf.7 ` [14:18:17] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [14:21:36] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10mmodell) a:03mmodell @aklapper: seems like a sane solution to me, I'll give it a shot. [14:28:13] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10mmodell) Updating edges didn't work due to duplicate key errors. Should I just delete the edges? [14:31:03] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10mmodell) `lang=sql MariaDB [phabricator_maniphest]> select count(*) from edge where dst LIKE "PHID-MLST-%... [14:33:29] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10mmodell) ok I deleted the destination edges, that should hopefully resolve this specific issue? [14:34:15] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10dcausse) >>! In T224448#5216526, @thcipriani wrote: >>>! In T224448#5216469, @dcausse wrote: >> Here SendEmail-1 is not "blocked" it's just waiting for jobs, ho... [14:37:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) Group0 at 1.34.0-wmf.7 ([[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/512878 | gerrit ]], [[ https:/... [14:48:28] 10Project-Admins, 10Phlogiston: Rename #Category to #Category-Phlogiston - https://phabricator.wikimedia.org/T224450 (10Jdforrester-WMF) Move it to be a sub-project of Phlog? [14:49:53] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani) >>! In T224448#5216643, @dcausse wrote: >>>! In T224448#5216526, @thcipriani wrote: >>>>! In T224448#5216469, @dcausse wrote: >>> Here SendEmail-1 i... [14:50:19] 10Phabricator, 10Phabricator-Bot-Requests: Request for conduit token of Phabricator_maintenance user - https://phabricator.wikimedia.org/T224099 (10mmodell) @ladsgroup: I can silence the bulk job [14:53:04] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Cmjohnson) [14:53:31] twentyafterfour: I make the list and prepare the job, and then I let you know when it's ready. My big problem is that the list changes basically every hour so I need to make sure it doesn't remove anything by mistake. [14:57:51] twentyafterfour: "phabricator/ $ ./bin/bulk make-silent --id 1795" let me know when it's done and I start it [15:18:27] (03CR) 10Eric Gardner: [C: 03+1] Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [15:37:09] Amir1: done [15:37:29] twentyafterfour: started [15:39:05] twentyafterfour: done, all of open tasks that don't need the tag are removed. Do you think I should the closed tasks too? That shouldn't be too hard [15:39:50] Amir1: that's up to you, I don't think the closed tasks are too important really [15:40:05] yeah, let's keep it like that [16:17:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10Tgr) {T224493} happened in the middle of the train period, any thoughts? [16:43:14] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Krinkle) > Production branch pruning needs a commit to delete. Can you elaborate? The link to branch prunin... [16:49:21] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10matmarex) 05Open→03Resolved Seems fine, thanks! [16:49:25] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [16:50:50] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5217787, @Krinkle wrote: >> Production branch pruning needs a commit to dele... [16:51:54] 10Phabricator, 10Phabricator-Bot-Requests, 10User-Ladsgroup: Request for conduit token of Phabricator_maintenance user - https://phabricator.wikimedia.org/T224099 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Okay, now the first batch is ran [17:00:14] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Krinkle) >>! In T223602#5217833, @Jdforrester-WMF wrote: >>>! In T223602#5217787, @Krinkle wrote: >>> Produc... [17:10:45] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix image var reference in teardown [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512408 (owner: 10Dduvall) [17:11:29] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512408 (owner: 10Dduvall) [17:11:31] (03Merged) 10jenkins-bot: pipeline: Fix image var reference in teardown [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512408 (owner: 10Dduvall) [17:12:36] (03CR) 10Dduvall: [C: 03+2] pipeline: Include image tags in Gerrit comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512409 (owner: 10Dduvall) [17:13:28] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512409 (owner: 10Dduvall) [17:13:30] (03Merged) 10jenkins-bot: pipeline: Include image tags in Gerrit comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512409 (owner: 10Dduvall) [17:13:52] (03CR) 10jenkins-bot: pipeline: Fix image var reference in teardown [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512408 (owner: 10Dduvall) [17:14:20] (03CR) 10jenkins-bot: pipeline: Include image tags in Gerrit comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512409 (owner: 10Dduvall) [17:16:36] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-rehearse" [blubber] - 10https://gerrit.wikimedia.org/r/511784 (owner: 10Dduvall) [17:39:06] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [17:45:41] (03PS1) 10Dduvall: pipeline: Fix default configuration for deploy.test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 [17:46:45] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 (owner: 10Dduvall) [17:47:01] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix default configuration for deploy.test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 (owner: 10Dduvall) [17:47:34] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 (owner: 10Dduvall) [17:47:36] (03Merged) 10jenkins-bot: pipeline: Fix default configuration for deploy.test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 (owner: 10Dduvall) [17:48:17] (03CR) 10jenkins-bot: pipeline: Fix default configuration for deploy.test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512937 (owner: 10Dduvall) [17:50:54] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-rehearse" [blubber] - 10https://gerrit.wikimedia.org/r/511784 (owner: 10Dduvall) [17:58:26] 10Gerrit, 10Jade, 10Repository-Admins: Can't update extensions.git because gerrit won't serve the commit at which the Jade extension is pointed? - https://phabricator.wikimedia.org/T224316 (10Jdforrester-WMF) 05Open→03Invalid I deleted and recloned extensions.git and it worked fine. Odd. [18:06:10] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Security, 10Upstream: Integrate MFA into Gerrit - https://phabricator.wikimedia.org/T198813 (10Framawiki) Just note that this task looks very important to me. I think it should really be solved in one way or in another. [18:12:30] (03PS1) 10Dduvall: pipeline: Fix teardown check and build result [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 [18:13:11] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:13:45] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix teardown check and build result [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:14:12] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:14:14] (03Merged) 10jenkins-bot: pipeline: Fix teardown check and build result [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:14:41] (03CR) 10jenkins-bot: pipeline: Fix teardown check and build result [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:32:02] (03CR) 10Jforrester: "recheck" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:32:50] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512944 (owner: 10Dduvall) [18:47:51] (03PS1) 10Dduvall: pipeline: Work around Groovy CPS choking on inner class type [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512953 [18:49:45] (03CR) 10Dduvall: [C: 03+2] pipeline: Work around Groovy CPS choking on inner class type [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512953 (owner: 10Dduvall) [18:50:25] (03Merged) 10jenkins-bot: pipeline: Work around Groovy CPS choking on inner class type [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512953 (owner: 10Dduvall) [18:50:49] (03CR) 10jenkins-bot: pipeline: Work around Groovy CPS choking on inner class type [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512953 (owner: 10Dduvall) [19:10:17] (03PS1) 10Umherirrender: [ConfirmEdit] Add Math as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512957 [19:13:25] (03PS1) 10Umherirrender: [Kartographer] Add GeoData as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512958 [20:06:09] (03PS1) 10Umherirrender: [VisualEditor] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/512972 [20:57:08] 10Release-Engineering-Team, 10Gerrit-Privilege-Requests: Enable Igor Kim to submit patches to wikidata/query/blazegraph - https://phabricator.wikimedia.org/T224521 (10Smalyshev) [20:58:47] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Phabricator: Clean up deadlocked apache processes - https://phabricator.wikimedia.org/T187790 (10Dzahn) removed again since we are not seeing the leaks anymore since our recent upgrade to stretch and phab1003 [21:01:35] brennen hi, wondering if you could add "Igor Kim" to https://gerrit.wikimedia.org/r/#/admin/groups/995,info please? (resolves https://phabricator.wikimedia.org/T224521) [21:02:31] (03PS1) 10Paladox: Cleanup perms for wikidata/query [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512978 [21:03:25] (03PS1) 10Paladox: Use wikidata-query over wikidata-query-blazegraph [wikidata/query/blazegraph] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512980 [21:05:44] paladox: looking [21:06:52] brennen thanks! [21:09:08] 10Release-Engineering-Team (Kanban), 10dev-images, 10local-charts, 10Patch-For-Review: Create fabfile.py for dev-images - https://phabricator.wikimedia.org/T223328 (10greg) p:05Triage→03Normal [21:09:11] i've cleaned up the perms in wikidata/query/blazegraph (https://gerrit.wikimedia.org/r/512980) and wikidata/query/rdf (https://gerrit.wikimedia.org/r/512978) [21:09:19] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Fix Install Script For Some Versions of Ubuntu - https://phabricator.wikimedia.org/T223715 (10greg) p:05Triage→03Normal [21:44:53] 10Phabricator, 10Release-Engineering-Team (Kanban): Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10Aklapper) Thanks! [22:12:48] (03CR) 10Smalyshev: [C: 03+1] Cleanup perms for wikidata/query [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512978 (owner: 10Paladox) [22:13:52] (03CR) 10Smalyshev: [C: 03+1] Use wikidata-query over wikidata-query-blazegraph [wikidata/query/blazegraph] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512980 (owner: 10Paladox) [22:50:34] 10Phabricator, 10observability, 10Patch-For-Review: don't page all of SRE for phabricator 'phd' service not running - https://phabricator.wikimedia.org/T224205 (10Dzahn) 05Open→03Resolved