[03:53:37] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<50.00%) [04:36:03] (03PS1) 10Samwilson: Add meetingtimes tool tests [integration/config] - 10https://gerrit.wikimedia.org/r/595356 [05:00:41] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Performance-Team: Enable GTID on beta cluster mariaDB once upgraded - https://phabricator.wikimedia.org/T139044 (10Marostegui) We have GTID enabled for replication on the slaves but we don't really use it for replication topology changes as we've f... [07:52:36] (03CR) 10Hashar: [C: 03+2] Add meetingtimes tool tests [integration/config] - 10https://gerrit.wikimedia.org/r/595356 (owner: 10Samwilson) [07:53:24] (03Merged) 10jenkins-bot: Add meetingtimes tool tests [integration/config] - 10https://gerrit.wikimedia.org/r/595356 (owner: 10Samwilson) [07:57:11] (03CR) 10Hashar: [C: 03+2] "Deployed ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/595356 (owner: 10Samwilson) [08:19:23] (03CR) 10Awight: [C: 03+1] "Looks useful!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/595276 (owner: 10Reedy) [08:27:52] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) [08:46:44] hashar hi, i'm getting "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset." on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/595479/ [08:49:01] paladox: rebasing needed [08:49:10] but also says "up 2 date" so yea.. [08:49:13] mutante it's a new patchset though [08:49:14] yeh [08:52:32] paladox: looking [08:52:39] thanks! [08:54:20] the repo is botched [08:54:34] 2020-05-11 08:42:14,807 DEBUG zuul.Repo: Updating repository /srv/zuul/git/operations/puppet [08:54:34] 2020-05-11 08:42:15,049 ERROR zuul.Merger: Unable to reset repo [08:54:39] cmdline: git fetch --tags -v origin [08:54:39] stderr: 'fatal: Could not read from remote repository. [08:54:46] that is on contint2001 [08:58:51] PROBLEM - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [09:00:39] PROBLEM - Check systemd state on contint2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:03:18] ^^^ stopped it [09:12:20] I dont understand the error [09:12:23] I can not reproduce it [09:12:24] :/ [09:13:50] i think it must have been caused by copying /var/lib/zuul over [09:13:58] because that happened just before [09:17:43] 10Phabricator: Have a way to auto-email saved phab queries on a regular interval - https://phabricator.wikimedia.org/T252301 (10RhinosF1) I've created some code that can generate the list, let me try and get it onto a tool and emailing me. [09:17:53] 10Phabricator, 10User-RhinosF1: Have a way to auto-email saved phab queries on a regular interval - https://phabricator.wikimedia.org/T252301 (10RhinosF1) a:03RhinosF1 [09:28:48] RhinosF1: i was wondering how the "opt-in" part would work [09:29:04] wouldn't it mean you have to send email to people to ask them if they want email? [09:29:23] or ping them on a ticket to ask if they want pings on that ticket [09:29:26] RECOVERY - zuul_merger_service_running on contint2001 is OK: PROCS OK: 1 process with regex args bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [09:29:39] ^ turns out it was messed up permissions from my rsync indeed [09:30:28] paladox: solved :) [09:30:37] great! [09:30:39] thanks! [09:31:51] mutante: they would ping on a ticket and I would add it to a config file [09:32:04] so you'd give me an email and the custom search [09:33:32] RhinosF1: aha, sounds a bit like review-bot. cool [09:33:35] thanks [09:35:15] mutante: is there any way to see if toolforge is queing the emails I'm spamming myself [09:39:18] RECOVERY - Check systemd state on contint2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:44:41] RhinosF1: i don't know, never used toolforge [09:44:56] i'm asking the cloud folks now [09:45:14] If I can mail to work properly, then I should be able to set the rest up easily [09:45:17] but in general if there is exim .. try "mailq" [10:33:42] mutante: it should be working by the end of today [10:35:30] 10Phabricator, 10User-RhinosF1: Have a way to auto-email saved phab queries on a regular interval - https://phabricator.wikimedia.org/T252301 (10RhinosF1) The tool functions. once I squash a bug that I’ve created fixed and unfixed, it will be ready. If anyone wants to use it, please send me how often you woul... [10:39:41] RhinosF1: to be honest, i will not be a user of this, but i see that separately [10:40:03] Ok [10:41:11] so just trying to help with a specific thing you wanted [10:41:35] in general i think Andre already went through that issue by getting "licked" tasks [10:41:52] and that's why that query was added [10:45:28] mutante: ah [10:47:07] * andre__ lacks context which "query" this is about [10:49:44] andre__: I am working on the phab saved query emails [10:50:03] I understand that, yes [10:50:12] Mutante was referring to the existing one on licked tags [10:50:23] but that is not a saved query. [10:50:32] instead that is an SQL one on the DB [10:50:48] No it isn’t. This covers anything phab search can. [10:52:25] Once I make the headers work again and set the cron running then it should be working. [10:54:43] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10LarsWirzenius) https://gerrit.wikimedia.org/r/admin/projects/pgp-public-keys [10:59:44] RhinosF1: My understanding was that you want to send yourself some saved query results via email for some reason. And allow other people to do the same thing when it comes to their saved queries. [11:01:45] yea, i meant the one that is executed by cron via script ..directly on the db [11:02:20] and just the part that it has already been tried to identify "licked" tasks [11:02:42] i saw some people unassign themselves after that campaign.afair [11:14:02] andre__: yep [11:14:08] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) A three step plan: * add sync-world, make sync emit a warning message * make a release * wait for a month * chang [11:14:25] The tool allows anyone to request an email is sent regularly with a saved query [11:15:03] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Build a Seakeeper MVP - https://phabricator.wikimedia.org/T248876 (10LarsWirzenius) p:05High→03Low [11:15:05] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Ask Legal if we can sign Argo's CLA - https://phabricator.wikimedia.org/T250152 (10LarsWirzenius) p:05Medium→03Low [11:15:13] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Create a threat model for new CI - https://phabricator.wikimedia.org/T240679 (10LarsWirzenius) p:05Medium→03Low [11:15:35] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) p:05Triage→03High [11:15:54] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) p:05High→03Medium [11:17:03] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Format Scap source code with black - https://phabricator.wikimedia.org/T252399 (10LarsWirzenius) [11:17:16] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Format Scap source code with black - https://phabricator.wikimedia.org/T252399 (10LarsWirzenius) p:05Triage→03Low [11:17:19] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Generate time.releng.team and people.json from one source of truth - https://phabricator.wikimedia.org/T250691 (10LarsWirzenius) p:05Low→03Lowest [11:20:02] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Python3-Porting: Port scap to Python 3.4 - https://phabricator.wikimedia.org/T246025 (10LarsWirzenius) [11:25:54] 10Phabricator, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10jhsoby-WMNO) 05Open→03Resolved a:03jhsoby-WMNO [11:26:01] 10Phabricator, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10jhsoby-WMNO) 05Resolved→03Open [11:42:29] 10Phabricator, 10Project-Admins, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10Dzahn) [11:42:39] 10Phabricator, 10Project-Admins, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10Dzahn) [11:42:44] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Dzahn) [11:43:47] 10Phabricator, 10Project-Admins, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10Aklapper) [11:43:53] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) [12:04:50] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [12:09:49] (03PS1) 10Hashar: Switch jobs to contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595511 (https://phabricator.wikimedia.org/T224591) [12:10:47] (03PS1) 10Hashar: fab: deploy zuul config on contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595512 (https://phabricator.wikimedia.org/T224591) [12:13:15] (03CR) 10Hashar: [C: 03+2] Switch jobs to contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595511 (https://phabricator.wikimedia.org/T224591) (owner: 10Hashar) [12:13:17] (03CR) 10Hashar: [C: 03+2] fab: deploy zuul config on contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595512 (https://phabricator.wikimedia.org/T224591) (owner: 10Hashar) [12:14:14] (03Merged) 10jenkins-bot: Switch jobs to contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595511 (https://phabricator.wikimedia.org/T224591) (owner: 10Hashar) [12:14:17] (03Merged) 10jenkins-bot: fab: deploy zuul config on contint2001 [integration/config] - 10https://gerrit.wikimedia.org/r/595512 (https://phabricator.wikimedia.org/T224591) (owner: 10Hashar) [12:16:36] 10Release-Engineering-Team, 10serviceops: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Cmjohnson) [12:29:04] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [12:42:50] 10Project-Admins, 10Inuka-Team: Clean up open #New-Readers tasks in Phabricator - https://phabricator.wikimedia.org/T251706 (10Aklapper) I've archived #New-Readers (though there are still a few tasks). Big Thanks for @AMuigai for the cleanup done so far! <3 [13:06:05] 10Phabricator, 10WMNO-General: Make #WMNO-General a subproject of #WMNO - https://phabricator.wikimedia.org/T221864 (10Aklapper) a:05jhsoby-WMNO→03None @jhsoby-WMNO: Sorry for the extremely late reply. Somehow I never got subscribed to this task (which puzzles me a bit). :( I guess this is still wanted? T... [13:18:55] 10Release-Engineering-Team, 10Wikimedia-GitHub, 10Developer-Advocacy (Apr-Jun 2020), 10User-Majavah: Automatically close Pull Requests in repos mirrored on Github - https://phabricator.wikimedia.org/T249703 (10Aklapper) Big Thanks to everybody who worked on this (especially Majavah for diving into this wit... [13:27:45] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dvorapa) p:05Triage→03Unbreak! Okay, now even merge fails for no reason: https://gerrit.wikimedia.... [13:31:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dzahn) p:05Unbreak!→03High A planned migration of the CI system is in progress. [13:36:44] Yippee, build fixed! [13:36:44] Project beta-code-update-eqiad build #296061: 09FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/296061/ [13:42:26] PROBLEM - jenkins_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [13:44:33] Yippee, build fixed! [13:44:34] Project beta-code-update-eqiad build #296061: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/296061/ [13:47:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dzahn) A revert back to the old system as before has happened due to issues with Jenkins after the swi... [13:48:42] 10Release-Engineering-Team, 10Repository-Admins, 10Patch-For-Review, 10Product-Analytics (Kanban): Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10nshahquinn-wmf) @mpopov should we not use a repo like `analytics/wmf-product/jobs`? We'll want rep... [13:49:22] the contint2001 alert is already gone in web UI [13:49:25] we reverted [14:02:22] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) The upgrade itself went well: * the rs... [14:05:23] 10Phabricator, 10User-RhinosF1: Have a way to auto-email saved phab queries on a regular interval - https://phabricator.wikimedia.org/T252301 (10RhinosF1) 05Open→03Resolved The script runs weekly on a Monday at 2pm UTC. I'm happy to setup other frequencies on request. Please fill out https://phabricator.w... [14:31:27] 10Release-Engineering-Team, 10decommission, 10serviceops: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10RobH) [14:38:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dvorapa) I see, I didn't know. I wish you the upgrade will work out finally. [14:47:14] 10Release-Engineering-Team, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) [14:47:20] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dzahn) @Dvorapa Sorry, we should have mailed more public venues like wikitech-l about this. We will do... [14:55:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10hashar) 05Open→03Declined CI was under maintenance (T224591) which I have announced Thursday on wi... [15:08:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Krinkle) Some services are maintained by their owners in both production data center... [15:43:53] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot: Pywikibot post-merge job fails to start up - https://phabricator.wikimedia.org/T252310 (10Dvorapa) I see, I thought wikitech list is for that wiki, but it was weird to me there is no WMCI, WM-... [15:46:50] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Core Platform Team, 10Quibble, 10Regression: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10Krinkle) [15:50:32] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the produ... - https://phabricator.wikimedia.org/T220398 [15:50:36] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Operations, and 7 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10thcipriani) [15:50:40] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10thcipriani) [15:54:37] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the produ... - https://phabricator.wikimedia.org/T220398 [15:54:42] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10thcipriani) [15:54:48] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 4 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10thcipriani) [15:54:59] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the produ... - https://phabricator.wikimedia.org/T220398 [15:57:17] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the produ... - https://phabricator.wikimedia.org/T220398 [16:06:18] 10Continuous-Integration-Config: Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches - https://phabricator.wikimedia.org/T90303 (10Krinkle) 05Open→03Resolved [16:06:25] 10Continuous-Integration-Config, 10Wikibase-JavaScript-Api, 10Wikidata: [Task] Make Wikibase-JavaScript-API testextension job run composer - https://phabricator.wikimedia.org/T110172 (10Krinkle) [16:06:28] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review: [Task] change phpunit CI job for Wikidata.org to use composer - https://phabricator.wikimedia.org/T107834 (10Krinkle) [16:06:31] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663 (10Krinkle) [16:21:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10thcipriani) [16:21:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10thcipriani) [16:24:24] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Prod-Kubernetes, and 2 others: TEC3:O6:O:6.1:Q3: Deployment Pipeline Documentation - https://phabricator.wikimedia.org/T213090 (10thcipriani) [16:57:28] 10Release-Engineering-Team-TODO, 10User-DannyS712, 10ci-test-error: Scap plugins in operations/mediawiki-config don't pass tox, so CI is disabled - https://phabricator.wikimedia.org/T248040 (10Jdforrester-WMF) [17:02:54] 10Phabricator, 10Wikimedia-Site-requests: Create a Phabricator form for requesting site requests - https://phabricator.wikimedia.org/T139844 (10Aklapper) [17:03:02] 10Phabricator, 10Wikimedia-Site-requests: Create a Phabricator form for requesting site requests - https://phabricator.wikimedia.org/T139844 (10Aklapper) Thanks @Urbanecm for pinging me on this one, and I apologize that this task has been lingering. I have added some more links / info / a bit of rephrase to t... [17:04:07] 10Phabricator, 10Wikimedia-Site-requests: Create a Phabricator form for requesting site configuration changes - https://phabricator.wikimedia.org/T139844 (10Aklapper) [17:19:40] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T249964 (10DannyS712) After the mess I made last time, I thought it would be helpful to list my patches against core that are going out with this tr... [17:27:29] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10The_Polish) One year from the malfunction, still not working. [17:29:34] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T249964 (10DannyS712) Hmm. Not sure if it will cause issues, but `RevisionItemBase::getId` currently returns string despite being documented to retu... [17:51:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Quibble: Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10Jdforrester-WMF) [17:51:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Quibble: Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10Jdforrester-WMF) [17:53:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Quibble: Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10Jdforrester-WMF) [17:53:13] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Create buster-based images for quibble - https://phabricator.wikimedia.org/T250514 (10Jdforrester-WMF) [17:53:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10serviceops: upgrade MediaWiki appservers to Debian 10 (buster) - https://phabricator.wikimedia.org/T245757 (10Jdforrester-WMF) [17:56:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble: Test MW code in buster as well as stretch - https://phabricator.wikimedia.org/T252434 (10Jdforrester-WMF) [17:56:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble: Test MW code in buster as well as stretch - https://phabricator.wikimedia.org/T252434 (10Jdforrester-WMF) [17:57:04] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Create buster-based images for quibble - https://phabricator.wikimedia.org/T250514 (10Jdforrester-WMF) [17:57:10] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Create buster-based images for quibble - https://phabricator.wikimedia.org/T250514 (10Jdforrester-WMF) [17:57:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Quibble: Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10Jdforrester-WMF) [18:09:06] 10Release-Engineering-Team, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 2 others: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) Timo suggested I just add routing to these backends in ATS via horizon hiera in... [18:18:44] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments, and 2 others: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) 05Open→03Resolved Appears stable on all wikis, but will continue monitoring. Thanks all... [18:49:32] it seems zuul is ignoring +2/gate pipeline for fresh.git - https://gerrit.wikimedia.org/r/#/c/fresh/+/591214/ [18:49:59] pipeline added as part of https://github.com/wikimedia/integration-config/commit/27086a3d019e94 [18:50:17] oh.. [18:50:22] the parent commit is pending [18:51:09] 2020-05-11 18:44:21,857 INFO zuul.Scheduler: Adding fresh, to [18:51:23] This is from /var/log/zuul on contint1001 [18:51:29] alright, mystery solved [18:54:43] Ha. [19:04:18] hm, looks like puppet is broken in deployment-prep [19:04:42] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute ... on at least 1 of the following 'server_urls': https://deployment-puppetdb03.deployment-prep.eqiad.wmflabs [19:05:29] The last Puppet run was at Thu Apr 30 21:45:25 UTC 2020 (15679 minutes ago). [19:05:35] for a long time! [19:06:02] puppetdb is dead [19:06:55] gonna try to start it [19:07:14] on deployment-puppetdb03 [19:09:01] seems better now? [19:09:01] does this channel have a !log? let's find uot! [19:09:12] oh maybe not better, hmm [19:09:17] facts retrieval works [19:09:32] ah now it is a syntax error oko that's better [19:09:52] !log started dead puppetdb service on deployment-puppetdb03 [19:09:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:17:19] !log set profile::trafficserver::tls::inbound_tls_settings.prioritize_chacha: 1, profile::cache::varnish::frontend::fe_vcl_config.large_objects_cutoff: 262144 in prefixpuppet deployment-cache-text horizon hiera to fix puppet on deployment-cache-text06 [19:17:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:22:46] thcipriani: your phab avatar is cracking me up [19:23:37] urandom: heh, yeah, my election photo: https://photos.tylercipriani.com/thumbs/8e/2ad9c55baee36d1528aaa8eee36f8e/large.jpg [19:24:15] it's missing an eagle [19:24:29] And a baseball bat. [19:26:47] how many symbols of america can I cram into one photo before it's too gauche? I feel like I'm already walking a fine line. [19:27:03] The wax jacket is a bit British. [19:27:18] Maybe it should have been a US football jersey? [19:27:50] https://en.wikipedia.org/wiki/Apple_pie#/media/File:Motherhood_and_apple_pie.jpg [19:28:27] haha, now I can just superimpose my face [19:33:14] 10Release-Engineering-Team, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 2 others: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) 05Open→03Resolved a:03Ottomata [20:01:07] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Blubber: For node images, run npm ci instead of npm install - https://phabricator.wikimedia.org/T250764 (10jeena) 05Open→03Resolved [20:01:51] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Blubber: For node images, run npm ci instead of npm install - https://phabricator.wikimedia.org/T250764 (10jeena) Deployed to production on May 8. [20:07:01] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Determine if it is possible to migrate remaining integration jessie permanent instances to stretch/buster - https://phabricator.wikimedia.org/T242909 (10jeena) 05Open→03Resolved All jobs h... [20:10:05] 10Continuous-Integration-Config, 10Fresh: Move Fresh to Gerrit (mirror to GitHub) - https://phabricator.wikimedia.org/T250766 (10Krinkle) [20:21:11] 10Project-Admins, 10Inuka-Team: Clean up open #New-Readers tasks in Phabricator - https://phabricator.wikimedia.org/T251706 (10AMuigai) @Aklapper Most of the tickets in that backlog have been taken care of. The remaining three are awaiting feedback. [20:22:26] 10Project-Admins, 10Inuka-Team: Clean up open #New-Readers tasks in Phabricator - https://phabricator.wikimedia.org/T251706 (10AMuigai) > Big Thanks for @AMuigai for the cleanup done so far! <3 Thanks for bringing the board to our attention! [20:25:25] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Performance-Team (Radar): Enable GTID on beta cluster mariaDB once upgraded - https://phabricator.wikimedia.org/T139044 (10aaron) [20:29:17] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Parsoid, 10Patch-For-Review: Undeploy ParsoidBatchAPI from the Wikimedia cluster - https://phabricator.wikimedia.org/T242430 (10ssastry) [20:31:56] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Parsoid, 10Patch-For-Review: Undeploy ParsoidBatchAPI from the Wikimedia cluster - https://phabricator.wikimedia.org/T242430 (10ssastry) >>! In T242430#5837321, @ssastry wrote: > @cscott pointed out that our developer standalone mode references the... [20:50:20] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Parsoid, 10Patch-For-Review: Undeploy ParsoidBatchAPI from the Wikimedia cluster - https://phabricator.wikimedia.org/T242430 (10Jdforrester-WMF) >>! In T242430#6126839, @ssastry wrote: > This is now resolved as part of T248340. Does that mean we c... [20:54:18] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Parsoid, 10Patch-For-Review: Undeploy ParsoidBatchAPI from the Wikimedia cluster - https://phabricator.wikimedia.org/T242430 (10ssastry) Any time that is convenient for you - there is no urgency on our part. [21:27:14] 10Release-Engineering-Team, 10Wikimedia-Extension-setup, 10CPT Initiatives (API Gateway): Determine how to host Chameleon skin and Bootstrap extension in gerrit while primary development happens in github - https://phabricator.wikimedia.org/T252463 (10CCicalese_WMF) [21:36:59] (03PS1) 10RhinosF1: Edit Project Config [labs/tools/phabsearchemail] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/595660 [21:37:23] (03Abandoned) 10RhinosF1: Edit Project Config [labs/tools/phabsearchemail] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/595660 (owner: 10RhinosF1) [21:39:42] 10Continuous-Integration-Infrastructure, 10Jenkins: 404 link in README file in integration/jenkins repository - https://phabricator.wikimedia.org/T252466 (10Aklapper) [21:48:14] 10Continuous-Integration-Config: Add Jenkins monitoring to new Page Exchange extension - https://phabricator.wikimedia.org/T252473 (10Yaron_Koren) [22:01:51] (03PS1) 10Krinkle: docs: Update outdated README [integration/jenkins] - 10https://gerrit.wikimedia.org/r/595661 (https://phabricator.wikimedia.org/T252466) [22:01:53] 10Continuous-Integration-Infrastructure, 10Jenkins: 404 link in README file in integration/jenkins repository - https://phabricator.wikimedia.org/T252466 (10Krinkle) The integration/jenkins,git repo is (almost) entirely unused. I've replaced the prominent link on that mediawiki.org page with a link to integra... [22:02:13] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: 404 link in README file in integration/jenkins repository - https://phabricator.wikimedia.org/T252466 (10Krinkle) p:05Triage→03Low a:03Krinkle [22:02:31] (03CR) 10jerkins-bot: [V: 04-1] docs: Update outdated README [integration/jenkins] - 10https://gerrit.wikimedia.org/r/595661 (https://phabricator.wikimedia.org/T252466) (owner: 10Krinkle) [22:09:27] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10EngProd-Virtual-Hackathon, 10Release Pipeline (Blubber): Allow new blubber builders to be implemented in yaml - https://phabricator.wikimedia.org/T201875 (10dduvall) [23:39:09] 10Continuous-Integration-Config, 10Fresh: Move Fresh to Gerrit (mirror to GitHub) - https://phabricator.wikimedia.org/T250766 (10Krinkle) 05Open→03Resolved [23:46:38] (03PS1) 10Krinkle: doc: Add entry for Fresh [integration/docroot] - 10https://gerrit.wikimedia.org/r/595711 (https://phabricator.wikimedia.org/T251213) [23:48:07] (03CR) 10Krinkle: [C: 03+2] doc: Add entry for Fresh [integration/docroot] - 10https://gerrit.wikimedia.org/r/595711 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [23:52:11] (03Merged) 10jenkins-bot: doc: Add entry for Fresh [integration/docroot] - 10https://gerrit.wikimedia.org/r/595711 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle)