[00:10:53] <wmf-insecte>	 Yippee, build fixed!
[00:10:53] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #11381: 09FIXED in 6 min 49 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/11381/
[00:14:16] <paladox>	 2.15 is offically end of life today :( (3.1 is currently being built)
[00:55:05] <wikibugs>	 10Phabricator, 10Research: Fix Research component tags to make them non-mutually-exclusive - https://phabricator.wikimedia.org/T238258 (10leila) >>! In T238258#5665862, @Aklapper wrote: > Thanks. I'm thinking of a partial revert by making #address-knowledge-gaps, #foundational, #Knowledge-Integrity independent...
[01:54:43] <wikibugs>	 (03CR) 1020after4: [C: 03+1] Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4)
[02:12:16] <wikibugs>	 10Deployments, 10Release-Engineering-Team-TODO, 10WMF-JobQueue, 10Core Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: Review removal of ukwikimedia wiki - https://phabricator.wikimedia.org/T218170 (10Dzahn) also see T164490
[04:05:00] <wikibugs>	 10Phabricator: Merge the Phabricator Priority values "Low" and "Lowest" - https://phabricator.wikimedia.org/T228759 (10Tgr) >>! In T228759#5357845, @Aklapper wrote: >> I'd prefer to rename `lowest` to `later` > Every task is `later` than now (as long as you cannot travel back in time). My immediate thought would...
[05:38:07] <apergos>	 1k batch of uploads/captions/depicts started, with 9 seconds between edits/uploads, just like yesterday
[06:25:50] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out)
[08:53:49] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Mainframe98) Today again; I had remained logged in since my last comment (10 days ago - T222472#563...
[09:00:45] <apergos>	 uploads are done, captions have started
[09:06:30] <wikibugs>	 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10GuillermoABrunner) Here they have shared some valuable information which is useful for the students who are struggling to prepare a quality assignment [[ https://businessays.net/business-communications-chapter6/ | http...
[09:10:52] <wikibugs>	 (03PS1) 10Legoktm: Pass CPPFLAGS onto tests to pick up proper include path for opencv4 [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551146
[09:10:54] <wikibugs>	 (03PS1) 10Legoktm: Avoid deprecated C-API constants [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551147
[09:10:56] <wikibugs>	 (03PS1) 10Legoktm: Always link against opencv_imgcodecs [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551148
[09:10:58] <wikibugs>	 (03PS1) 10Legoktm: Bump version to 1.4.0 [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551149
[09:15:07] <wikibugs>	 (03PS2) 10Legoktm: Pass CPPFLAGS onto tests to pick up proper include path for opencv4 [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551146 (https://phabricator.wikimedia.org/T216864)
[09:15:09] <wikibugs>	 (03PS2) 10Legoktm: Avoid deprecated C-API constants [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551147 (https://phabricator.wikimedia.org/T216864)
[09:15:11] <wikibugs>	 (03PS2) 10Legoktm: Always link against opencv_imgcodecs [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551148 (https://phabricator.wikimedia.org/T216864)
[09:15:13] <wikibugs>	 (03PS2) 10Legoktm: Bump version to 1.4.0 [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/551149
[09:15:56] <paladox>	 The post above that looks like spam?
[09:15:58] <paladox>	 From 9:04am
[09:16:12] <paladox>	 *9:06
[09:50:19] <apergos>	 yeah looks like a spammer indeed
[09:50:26] <apergos>	 just two comments but still
[12:29:58] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10LarsWirzenius) I had to log in today as well. I had been logged in since October 24, so it seems th...
[12:46:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Krenair) @fgiunchedi Do we need to do anything else to get rid of deployment-logstash2 and use deployment-logstash03 in...
[12:49:54] <wikibugs>	 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Krenair) @Eevans: It has been 6 months, please respond.
[13:48:51] <apergos>	 captions have long since finished and there are about 600 depicts to go
[14:20:48] <wikibugs>	 10Continuous-Integration-Config, 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10Osnard) Thanks for your feedback, I see your concerns. To be honest, I am also not very happy with the `REL1_31_dev`-branch-approach. But...
[14:32:33] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Anomie) Ran into what is probably this this morning while trying to investigate T238253 and T...
[14:43:02] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Wikimedia-Technical-Conference-2019: Unconference: [Workshop] CI counselling session - https://phabricator.wikimedia.org/T238275 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF
[15:03:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10User-DannyS712: BetaCluster: ExternalStoreException - Unable to store text to external storage - https://phabricator.wikimedia.org/T228088 (10Anomie) Unfortunately due to {T233134} I can't seem to find any logs since September that might point to an underlying issue. There are...
[15:48:38] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[15:58:34] <greg-g>	 who pushed a bunch of ops/puppet changes? :) ^
[15:59:42] <greg-g>	 (brandon)
[15:59:44] <James_F>	 greg-g: Brandon, see -operations
[16:01:44] <apergos>	 depicts are done, that's everything for today
[16:02:03] <apergos>	 I doubt I'll run anythig over the weekend, if I do it will be announced here though
[16:03:18] <James_F>	 apergos: Cool.
[16:05:15] <wikibugs>	 10Release-Engineering-Team-TODO, 10MediaWiki-General, 10MW-1.35-release, 10PHP 7.2 support, 10Upstream: PHP 7.2 garbage collector segfault - https://phabricator.wikimedia.org/T228346 (10Jdforrester-WMF) 05Open→03Stalled
[16:11:17] <Lucas_WMDE>	 …is anything actually running in Zuul right now?
[16:11:24] <Lucas_WMDE>	 no matter what I click, everything’s queued
[16:11:43] <Lucas_WMDE>	 ah, no, 550940,5 (operations/puppet) is actually running ^^
[16:12:42] <James_F>	 It is surprisinglystuck stuck.
[16:13:49] <Lucas_WMDE>	 looking better now
[16:16:19] <James_F>	 Not much better.
[16:19:29] <James_F>	 !log Gearman having fun. Stuck with 1233 merge changes.
[16:19:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:20:46] <paladox>	 heh
[16:21:39] <James_F>	 Now down to 1222.
[16:23:18] <paladox>	 Error in GET /r/changes/?q=20130529000000&n=25&O=81
[16:23:29] <paladox>	 i dunno how someone managed to get that largeeeee number :P
[16:27:12] <Lucas_WMDE>	 if the # of waiting jobs in https://w.wiki/C9F continues to decrease at its current pace, it’ll be done in a bit over 3 hours
[16:29:07] <James_F>	 Cartesian product of too many patches.
[16:30:16] <James_F>	 hashar did a hack which is dropping it quickly
[16:31:56] <James_F>	 Down to 900.
[16:39:36] <James_F>	 Now below 250.
[16:41:10] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:45:20] <James_F>	 !log Gearman merge queue clear thanks to hashar, all fixed.
[16:45:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:53:36] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Wikimedia-Technical-Conference-2019: Unconference: [Workshop] CI counselling session - https://phabricator.wikimedia.org/T238275 (10hashar) That went well overall, we got some useful feedback and calls for actions.
[16:53:43] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10International-Developer-Events, 10Wikimedia-Technical-Conference-2019, and 2 others: Wikimedia Technical Conference 2019 Session: System level testing: patterns an... - https://phabricator.wikimedia.org/T234635
[17:22:37] <James_F>	 Phabricator seems very slow?
[17:23:03] <subbu>	 James_F, yes, for me as well.
[17:23:04] <paladox>	 yeh it's slow
[17:23:30] <James_F>	 hashar is looking now.
[17:23:47] <hashar>	 nothing I can see on the phab hosts monitoring dashboards
[17:23:49] <hashar>	 so no clue
[17:24:07] <hashar>	 I gues the backend is down somehow
[17:24:10] <James_F>	 Just got a "upstream request timeout" report.
[17:24:23] <hashar>	 beside twentyafterfour , not sure who can fix it!?
[17:24:48] <James_F>	 Yeah. :-(
[17:25:35] <paladox>	 php-fpm
[17:25:44] <paladox>	 check php-fpm :)
[17:26:32] <James_F>	 Now upgraded to a Varnish 503.
[17:27:14] <hashar>	 lets follow up in -operations channel
[17:27:20] <hashar>	 seems some SREs are looking into it
[17:27:48] <James_F>	 Kk.
[17:47:54] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused
[17:47:54] <shinken-wm>	 PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused
[18:38:19] <paladox>	 https://gerrit-review.googlesource.com/c/homepage/+/245756
[19:34:22] <wikibugs>	 10Phabricator, 10Operations: List of recent most active Phab "Priority" field setters - https://phabricator.wikimedia.org/T235153 (10Dzahn) a:03Dzahn
[19:36:20] <wikibugs>	 10Phabricator, 10Operations: List of recent most active Phab "Priority" field setters - https://phabricator.wikimedia.org/T235153 (10Dzahn) @Aklapper  here it is:  {P9646}
[19:36:29] <wikibugs>	 10Phabricator, 10Operations: List of recent most active Phab "Priority" field setters - https://phabricator.wikimedia.org/T235153 (10Dzahn) 05Open→03Resolved
[20:32:00] <wikibugs>	 10Release-Engineering-Team-TODO, 10MediaWiki-General, 10MW-1.35-release, 10PHP 7.2 support, 10Upstream: PHP 7.2 garbage collector segfault - https://phabricator.wikimedia.org/T228346 (10Reedy) >>! In T228346#5644458, @Jdforrester-WMF wrote: > In that case we'll want to bump to 7.2.22 in 1.35 when Parsoid...
[20:59:35] <wikibugs>	 10Phabricator, 10Release-Engineering-Team-TODO, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10Aklapper) `          for p in rules:              if p['rule'] == "PhabricatorPolicyRuleProjects":                  allowedProjects = p['value...
[21:52:19] <wikibugs>	 10Continuous-Integration-Config, 10Wikimedia-Technical-Conference-2019: Unconference: [Workshop] I'll help you replace Travis with Actions for GitHub-mastered repos - https://phabricator.wikimedia.org/T238244 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Done! Happy to help others and other r...
[21:53:17] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: Update PHP 7.2 image to 7.2.24 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/550919 (https://phabricator.wikimedia.org/T237275) (owner: 10Jforrester)
[21:54:04] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: Update PHP 7.2 image to 7.2.24 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/550919 (https://phabricator.wikimedia.org/T237275) (owner: 10Jforrester)
[21:56:13] <James_F>	 !log Docker: Adding new images refreshed for PHP 7.2.24 T237275
[21:56:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:56:16] <stashbot>	 T237275: Upgrade PHP 7.2.x CI images to PHP 7.2.24 to align with production - https://phabricator.wikimedia.org/T237275
[22:35:35] <wikibugs>	 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10mmodell) So how useful is the notification from gerritbot? I don't rely on it for an...
[23:01:05] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10DBA, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) Some scenarios that we should describe and test:  1. A simple failure...
[23:24:57] <wikibugs>	 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10Aklapper) @mmodell: Does "notification" only refer to creation of a comment in a Pha...