[02:48:42] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Developer Productivity, 10Epic: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Krinkle)
[05:20:42] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[05:26:18] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[05:59:41] <wikibugs>	 (03PS6) 10Kosta Harlan: Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072)
[06:08:31] <wikibugs>	 (03PS5) 10Kosta Harlan: Add install script to work around shell issues [releng/dev-images] - 10https://gerrit.wikimedia.org/r/584111 (https://phabricator.wikimedia.org/T246953)
[06:12:56] <wikibugs>	 (03CR) 10Kosta Harlan: Provide support for setting default configuration in dev image (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) (owner: 10Kosta Harlan)
[06:34:01] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%)
[06:44:01] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)
[09:32:25] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10jbond) >>! In T243808#6024187, @Dzahn wrote: > Alright. This is a one-time installation though to test the Gerrit upgrade to 2.16 and then remove i...
[10:10:48] <wikibugs>	 10Project-Admins: Create project Articles Needing links - https://phabricator.wikimedia.org/T249316 (10Urbanecm)
[10:15:00] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-webperf11 is CRITICAL: (Service Check Timed Out)
[10:19:50] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-webperf11 is OK: OK: All targets OK
[10:28:44] <wikibugs>	 10VPS-project-codesearch, 10Analytics: Add analytics/* gerrit repos to code search - https://phabricator.wikimedia.org/T249318 (10Addshore)
[12:01:21] <wikibugs>	 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Developer Productivity, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10LarsWirzenius) @Krinkle do you know when this started happening?   The actual change in Scap happened last year...
[12:10:51] <mutante>	 Phabricator says "Your browser timezone (UTC-4) differs from your profile timezone (UTC-7). You can ignore this conflict or adjust your profile setting to match your client."
[12:11:04] <mutante>	 that's fine but all the timezones i can select are starting with "America" 
[12:12:54] <mutante>	 when i go via profile->manage i can select them all though
[13:31:53] <wikibugs>	 10Release-Engineering-Team-TODO, 10Developer-Advocacy (Apr-Jun 2020), 10Documentation: Wikimedia Technical Talks: April 2020 - Basic of cryptography using OpenPGP and GnuPG - https://phabricator.wikimedia.org/T244813 (10LarsWirzenius) Given April 22 is now going to be a WMF holiday, will the talk be cancelle...
[14:25:02] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%)
[14:29:50] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%)
[14:35:00] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)
[14:39:50] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK
[14:40:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10ContentSecurityPolicy, and 3 others: .mp4 build artifacts not viewable due to CSP in Chrome - https://phabricator.wikimedia.org/T245658 (10hashar) 05Ope...
[14:44:23] <wikibugs>	 10Release-Engineering-Team-TODO, 10Developer-Advocacy (Apr-Jun 2020), 10Documentation: Wikimedia Technical Talks: April 2020 - Basic of cryptography using OpenPGP and GnuPG - https://phabricator.wikimedia.org/T244813 (10srodlund) @LarsWirzenius I just sent you an email before I saw this. It is really up to y...
[14:45:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10phpunit-patch-coverage, and 4 others: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10hash...
[14:48:54] <wmf-insecte>	 Yippee, build fixed!
[14:48:55] <wmf-insecte>	 Project beta-scap-eqiad build #294618: 09FIXED in 4 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/294618/
[14:55:22] <James_F>	 hashar: Did you restart release-jenkins with the CSP too?
[15:35:30] <hashar>	  James_F yes sir
[15:35:35] <hashar>	 let me double check
[15:35:55] <hashar>	 yes it has
[15:36:06] <hashar>	 checked with   ssh releases1001.eqiad.wmnet ps -u jenkins f
[15:36:25] <wikibugs>	 10RelEng-Archive-FY201718-Q1, 10Wikimedia-Developer-Summit, 10User-greg: Facilitate Wikidev'17 main topic "How to manage our technical debt" - https://phabricator.wikimedia.org/T147937 (10Aklapper)
[15:36:44] <James_F>	 hashar: Nice.
[15:49:24] <wikibugs>	 10Project-Admins, 10Design-Research: Using Tags (yellow) for Pragmatic Personas - https://phabricator.wikimedia.org/T124729 (10Aklapper) These six tags have been used by #Design-Research [exactly six times](https://phabricator.wikimedia.org/maniphest/query/D4GJfxg1.W7d/#R). Looks like this did not make much se...
[15:56:27] <wikibugs>	 10Project-Admins, 10Structured-Data-Backlog: Clarify confusion between Multimedia and Structured Data team and update docs accordingly - https://phabricator.wikimedia.org/T240281 (10dr0ptp4kt) > Would it make sense to identify those that are not and mark them accordingly?  @Jrbranaa you thinking like a new col...
[16:02:15] <James_F>	 hashar: Meeting?
[16:02:27] <hashar>	 yeah
[16:20:57] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] Add phpunit-standalone, for phpunit --group Standalone [integration/quibble] - 10https://gerrit.wikimedia.org/r/577638 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester)
[16:34:13] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quality-and-Test-Engineering-Team (QTE), 10CPT Initiatives (API Integration Tests), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10Jdforrester-WMF)...
[16:36:33] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Zuul, 10Developer Productivity: Abort a Zuul pipeline when one job completed with failures (change zuul scheduler's failure check... - https://phabricator.wikimedia.org/T248531
[16:36:35] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Scap (Scap3-Adoption-Phase2), 10Technical-Debt: Migrate deployment of integration/config for zuul to scap deploy - https://phabricator.wikimedia.org/T129357 (10Jdforrester-WMF)
[16:39:17] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 (10Jdforrester-WMF) Per @hashar, it's not running on the `/` mount point but in another p...
[16:41:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 (10Lucas_Werkmeister_WMDE) (For what it’s worth, I don’t remember encountering this error...
[16:43:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089 (10Jdforrester-WMF) Is there anything we in RelEng can do to help with this...
[16:45:24] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10hashar) The idea behind the proposed implementation would be covered by the new CI work.
[16:50:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Test-Coverage: Have a CI test that fails if a Beta Cluster extension has an unsatisfiable dependency - https://phabricator.wikimedia.org/T231601 (10Jdforrester-WMF) a:03...
[16:52:34] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089 (10colewhite) @Jdforrester-WMF Absolutely!  If someone would be willing to...
[17:21:29] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quality-and-Test-Engineering-Team (QTE), 10CPT Initiatives (API Integration Tests), and 2 others: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10zeljkofilipin)
[17:25:27] <wikibugs>	 (03CR) 10Will Doran: deployment-calendar: Add train triage on Wednesday (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/584640 (owner: 10Thcipriani)
[17:38:51] <hashar>	 gerrit on gerrit1002 is flapping. It starts and dies constantly
[17:38:59] <hashar>	 I noticed that while digging in icinga logs
[17:39:05] <hashar>	 and that is left unnoticed apparently :-\
[17:39:40] <hashar>	 Apr 02 20:39:50 gerrit1002 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 739926.
[17:40:13] <hashar>	 oh it does not have access to the db
[17:40:24] <hashar>	 not a big deal
[17:40:53] <paladox>	 oh, mutante ^
[17:48:36] <mutante>	 gerrit1002 is the temp test server for 2.16 upgrade, not production
[17:49:05] <mutante>	 monitoring should be in downtime 
[17:49:22] <mutante>	 and was last time i checked..with reason "testing"
[17:50:57] <mutante>	 yes, it does not have db access. which is good, but if it's blocking then let us know
[18:27:09] <paladox>	 mutante it should have db access (but only read, not write)
[18:29:02] <mutante>	 paladox: ok, it should be clarified there if possible
[18:29:11] <mutante>	 we gotta still fix the db_user i guesss
[18:29:38] <mutante>	 bbl for now
[18:29:42] <wikibugs>	 (03PS3) 10Jeena Huneidi: Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916)
[18:29:55] <paladox>	 mutante i think we've given the correct user to the server, but is it using the correct password? Or does the ip need to be added to the whitelist.
[18:30:04] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10User-brennen: Investigate duration of scap sync for 1.35.0-wmf.23 train deploy - https://phabricator.wikimedia.org/T247426 (10brennen) Dug into logstash with @thcipriani a bit last friday...
[18:36:17] <wikibugs>	 (03PS4) 10Jeena Huneidi: Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916)
[18:37:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916) (owner: 10Jeena Huneidi)
[18:37:46] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10mmodell) I don't think there is any system to automatically create phab repositories when a new gerrit...
[18:47:45] <wikibugs>	 (03PS5) 10Jeena Huneidi: Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916)
[18:52:51] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) Just to be clear, I don't need that to happen automatically. I just need to know if I did som...
[18:54:29] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916) (owner: 10Jeena Huneidi)
[18:54:36] <wikibugs>	 10Continuous-Integration-Config, 10Code-Health, 10Readers-Web-Backlog (Kanbanana-2019-20-Q4), 10Vue.js: Configure ESLint for Vue.js development - https://phabricator.wikimedia.org/T249304 (10Jdforrester-WMF) a:03Catrope Roan's working on this, with support from me and Ed. See https://github.com/wikimedia...
[18:55:27] <wikibugs>	 (03Merged) 10jenkins-bot: Add publish pipeline for helloworldoid [integration/config] - 10https://gerrit.wikimedia.org/r/584701 (https://phabricator.wikimedia.org/T248916) (owner: 10Jeena Huneidi)
[19:02:06] <longma>	 !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/584701
[19:02:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:03:27] <wikibugs>	 10Continuous-Integration-Config, 10Code-Health, 10Readers-Web-Backlog (Kanbanana-2019-20-Q4), 10Vue.js: Configure ESLint for Vue.js development - https://phabricator.wikimedia.org/T249304 (10Niedzielski) a:05Catrope→03None Thanks for looking out, @Jdforrester-WMF! I've linked to [[ https://wikimedia.sl...
[19:56:00] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%)
[19:58:29] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add phpunit-standalone, for phpunit --group Standalone [integration/quibble] - 10https://gerrit.wikimedia.org/r/577638 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester)
[19:58:42] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "We talked about it over a meeting." [integration/quibble] - 10https://gerrit.wikimedia.org/r/577638 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester)
[19:59:28] <wikibugs>	 (03Merged) 10jenkins-bot: Add phpunit-standalone, for phpunit --group Standalone [integration/quibble] - 10https://gerrit.wikimedia.org/r/577638 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester)
[20:02:47] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%)
[20:03:10] <James_F>	 hashar: Thanks!
[20:03:31] <James_F>	 hashar: Next, https://gerrit.wikimedia.org/r/c/integration/quibble/+/550087 :-)
[20:06:03] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)
[20:11:33] <wikibugs>	 (03CR) 10Brennen Bearnes: [C: 03+2] Provide support for setting default configuration in dev image (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) (owner: 10Kosta Harlan)
[20:12:48] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK
[20:23:54] <wikibugs>	 (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) (owner: 10Kosta Harlan)
[20:24:33] <wikibugs>	 (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] Add install script to work around shell issues [releng/dev-images] - 10https://gerrit.wikimedia.org/r/584111 (https://phabricator.wikimedia.org/T246953) (owner: 10Kosta Harlan)
[20:25:12] <wikibugs>	 (03PS6) 10Brennen Bearnes: Add install script to work around shell issues [releng/dev-images] - 10https://gerrit.wikimedia.org/r/584111 (https://phabricator.wikimedia.org/T246953) (owner: 10Kosta Harlan)
[20:25:18] <wikibugs>	 (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] Add install script to work around shell issues [releng/dev-images] - 10https://gerrit.wikimedia.org/r/584111 (https://phabricator.wikimedia.org/T246953) (owner: 10Kosta Harlan)
[20:30:16] <brennen>	 !log Updating dev-images docker-pkg files on contint1001 for T246953
[20:30:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:30:19] <stashbot>	 T246953: Docker MW installation command line doesn't work on Windows command shell or PowerShell - https://phabricator.wikimedia.org/T246953
[20:32:19] <wikibugs>	 (03PS5) 10Jforrester: Add pattern based fallback for branches [integration/quibble] - 10https://gerrit.wikimedia.org/r/550087 (owner: 10Robert Vogel)
[20:40:29] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quality-and-Test-Engineering-Team (QTE), 10CPT Initiatives (API Integration Tests), and 2 others: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel) >>! In T248683#602...
[20:42:27] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quality-and-Test-Engineering-Team (QTE), 10CPT Initiatives (API Integration Tests), and 2 others: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10daniel)
[21:52:07] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10User-brennen: Investigate duration of scap sync for 1.35.0-wmf.23 train deploy - https://phabricator.wikimedia.org/T247426 (10thcipriani) >>! In T247426#6027489, @brennen wrote: > I have m...