[03:30:48] <wmf-insecte>	 Project beta-scap-eqiad build #241330: 04FAILURE in 8 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241330/
[03:42:02] <wmf-insecte>	 Yippee, build fixed!
[03:42:02] <wmf-insecte>	 Project beta-scap-eqiad build #241331: 09FIXED in 9 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241331/
[07:40:39] <wmf-insecte>	 Project beta-scap-eqiad build #241353: 04FAILURE in 7.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241353/
[07:48:28] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Task created via Conduit is reported in mail as "Reopened" - https://phabricator.wikimedia.org/T88005 (10mmodell) 05Open→03Resolved a:03mmodell
[07:48:46] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Task created via Conduit is reported in mail as "Reopened" - https://phabricator.wikimedia.org/T88005 (10mmodell) a:05mmodell→03epriestley
[07:54:00] <wmf-insecte>	 Yippee, build fixed!
[07:54:00] <wmf-insecte>	 Project beta-scap-eqiad build #241354: 09FIXED in 9 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241354/
[07:54:56] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0]
[07:58:19] <wikibugs>	 10Beta-Cluster-Infrastructure: Cannot access beta cluster db - https://phabricator.wikimedia.org/T217938 (10mmodell) mariadb no longer puts its  socket in /tmp/mysql.sock, it seems that the socket is in a 'systemd-private-xxx' directory within /tmp. See:  `lang=shell-session  twentyafterfour@deployment-db05:~$ l...
[08:20:41] <wmf-insecte>	 Project beta-update-databases-eqiad build #32411: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32411/
[08:45:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[08:54:21] <wikibugs>	 10Release-Engineering-Team (Next), 10CX-cxserver, 10Release Pipeline, 10serviceops, and 3 others: Migrate cxserver to kubernetes - https://phabricator.wikimedia.org/T213195 (10jijiki)
[09:21:12] <wmf-insecte>	 Yippee, build fixed!
[09:21:13] <wmf-insecte>	 Project beta-update-databases-eqiad build #32412: 09FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32412/
[09:41:30] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Switch change-propagation to the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/496387 (https://phabricator.wikimedia.org/T213193)
[09:53:43] <wikibugs>	 (03PS1) 10Hashar: jjb: look and capture 'core' file in Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[09:54:55] <hashar>	 !log ci: live hacked job https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/ in attempt to capture 'core' files from hhvm | https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ | T216689
[09:54:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:54:58] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[10:03:48] <hashar>	 oh my god
[10:07:23] <wikibugs>	 (03CR) 10Hashar: "It is never going to find anything since /workspace/src is in a Docker overlay and not in job workspace :/" [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[10:14:46] <wikibugs>	 (03PS2) 10Hashar: jjb: look and capture 'core' file in Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[10:22:54] <hashar>	 and the kernel has .... /proc/sys/kernel/core_pattern:/var/tmp/core/core.%h.%e.%p.%t
[10:23:00] <hashar>	 grbmblblblbl
[10:24:57] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0]
[10:38:37] <hashar>	 OH MY GOD
[10:39:09] <hashar>	 thcipriani: it is official I hate Docker.   Running PHPUnit tests with the source code in the Docker container overlay  eventually segfaults at some point
[10:39:48] <hashar>	 thcipriani: but mounting the source code from the host no more segfault (  mkdir src && chmod 2777 src && docker run --volume "$(pwd)/src":/src
[10:39:49] <hashar>	 ;D
[10:49:23] <hashar>	 !log triggering tests for all ContentTranslation pending changes # T216689
[10:49:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:49:26] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[12:06:39] <wikibugs>	 (03PS3) 10Hashar: jjb: Quibble jobs to capture core files if any [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[12:12:14] <hashar>	 !log Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them | https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/3 # T216689
[12:12:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:12:17] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[12:13:26] <wikibugs>	 (03CR) 10Lars Wirzenius: [C: 03+1] "Looks good to me." [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[12:31:47] <wikibugs>	 (03PS4) 10Hashar: jjb: Quibble jobs to capture core files if any [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[12:31:54] <hashar>	 !log Updated quibble-vendor-mysql-hhvm-docker to hopefully allow core dumps and capture them | https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/4 # T216689
[12:31:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:31:57] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[12:32:36] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: Quibble jobs to capture core files if any [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[12:37:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[12:40:31] <wikibugs>	 (03PS5) 10Hashar: jjb: Quibble jobs to capture core files if any [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[12:42:11] <wikibugs>	 (03PS6) 10Hashar: jjb: Quibble jobs to capture core files if any [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689)
[12:43:34] <Nikerabbit>	 12:13:57 Build step 'Execute a set of scripts' changed build result to FAILURE
[12:44:04] <Nikerabbit>	 Is this related to some recent change? Noticed it in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/496395
[12:48:18] <wikibugs>	 (03CR) 10Lars Wirzenius: [C: 03+1] "Assuming this works, LGTM" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[12:56:08] <wikibugs>	 (03CR) 10Hashar: jjb: Quibble jobs to capture core files if any (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[13:01:20] <wikibugs>	 10Release-Engineering-Team (Next), 10CX-cxserver, 10Release Pipeline, 10serviceops, and 3 others: Migrate cxserver to kubernetes - https://phabricator.wikimedia.org/T213195 (10jijiki) K8s is service currently ~8% of total traffic, we will rump it up to 50% tomorrow, please ping us if there are any issues
[13:01:25] <hashar>	 so
[13:01:26] <hashar>	 now
[13:15:15] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) 1.33.0-wmf.21 everywhere ([[ https://tools.wmflabs.org/sal/log/AWl8UuWLA1BDhGjCSszy | sal ]], [[ https://gerrit.wikimedi...
[13:40:27] <wikibugs>	 (03CR) 10Ppchelko: Switch change-propagation to the pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/496387 (https://phabricator.wikimedia.org/T213193) (owner: 10Alexandros Kosiaris)
[13:56:45] <hauskatze>	 Krenair: are we still using deployment-deploy01 for mysql? https://phabricator.wikimedia.org/P8200
[14:05:45] <wikibugs>	 10Beta-Cluster-Infrastructure: Cannot access beta cluster db - https://phabricator.wikimedia.org/T217938 (10MarcoAurelio) I am having the same issue: {P8200}  However I note that in the past I could do `sql eswiki;` on `deployment-deploy01` just fine.
[14:19:50] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Disable/remove Jenkins Jobs for Android app - https://phabricator.wikimedia.org/T198862 (10Dbrant) 05Open→03Resolved a:03Dbrant > we probably can remove the android jenkins plug...
[14:25:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[14:52:02] <hauskatze>	 issues with phab?
[14:52:04] <hauskatze>	 Unable to establish a connection to any database host (while trying "phabricator_policy"). All masters and replicas are completely unreachable. AphrontConnectionQueryException: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections.
[14:52:08] <hauskatze>	 looks intermitent though
[14:56:37] <mutante>	 ^ reported to DBAs
[15:00:04] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Paste search shows wrong length - https://phabricator.wikimedia.org/T140324 (10Aklapper) 05Open→03Resolved Thanks for re-checking. Yes, this seems fixed looking at https://phabricator.wikimedia.org/paste/query/jEgILY_JTKSZ/#R for the testcase given above.
[15:15:13] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10Jdlrobson)
[15:15:28] <jdlrobson>	 @paladox following up on our conversation yesterday ^
[15:15:39] <jdlrobson>	 i'm not sure who to route this problem too though
[15:17:00] <paladox>	 hmm
[15:17:13] <paladox>	 i have managed to get cors working once but forgot the change i did
[15:17:26] <paladox>	 but
[15:17:30] <paladox>	 CSP is different.
[15:19:19] <paladox>	 though phabricator.wikimedia.org is behind varnish so maybe there was a change to CSP for varnish?
[15:20:00] <paladox>	 yup
[15:20:03] <paladox>	 it's CSP jdlrobson 
[15:20:04] <paladox>	 content-security-policy: default-src https://phab.wmfusercontent.org; img-src https://phab.wmfusercontent.org data:; style-src https://phab.wmfusercontent.org 'unsafe-inline'; script-src https://phab.wmfusercontent.org; connect-src 'self'; frame-src 'self' https://commons.wikimedia.org; frame-ancestors 'none'; object-src 'none'; form-action 'self'; base-uri 'none'
[15:20:14] <paladox>	 need to have gerrit.w.org added to it
[15:33:28] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[15:36:01] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10chasemp)
[15:51:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[15:51:09] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[15:52:59] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:54:27] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0]
[15:57:24] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[16:05:14] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10herron) There looks to be a significant increase (about 1.5 million in the past hour) of log messages from the mediawiki "deprecated" c...
[16:12:02] <wikibugs>	 10Beta-Cluster-Infrastructure: Cannot access beta cluster db - https://phabricator.wikimedia.org/T217938 (10mmodell) on `deployment-deploy01` in `/usr/local/bin/sql` we have `php=php7.0` but apt doesn't have a `php7.0-redis package`
[16:12:33] <wikibugs>	 10Beta-Cluster-Infrastructure, 10PHP 7.0 support: Cannot access beta cluster db - https://phabricator.wikimedia.org/T217938 (10mmodell)
[16:13:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[16:19:35] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10Jdforrester-WMF) Argh, caused by ParsoidBatchAPI which I didn't spot. Patch coming.
[16:28:04] <shinken-wm>	 PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5)
[16:28:20] <wikibugs>	 (03PS1) 10Mholloway: Add Extension:WikimediaEditorTasks [tools/release] - 10https://gerrit.wikimedia.org/r/496484 (https://phabricator.wikimedia.org/T218136)
[16:34:17] <hashar>	 !log rollback quibble-vendor-mysql-hhvm-docker job to no more capture core files, we have enough and a good lead ( reverting https://gerrit.wikimedia.org/r/#/c/integration/config/+/496392/ ) # T216689
[16:34:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:34:20] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[16:35:08] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "I have only updated quibble-vendor-mysql-hhvm-docker and reverted back.  This patch would end up filling /tmp on hosts and I got enough tr" [integration/config] - 10https://gerrit.wikimedia.org/r/496392 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar)
[16:48:13] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10Aklapper) I'm not sure yet what to make out of {T218310} (missing exact repro steps, how often it happens, etc) b...
[16:52:00] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+1] Add Extension:WikimediaEditorTasks [tools/release] - 10https://gerrit.wikimedia.org/r/496484 (https://phabricator.wikimedia.org/T218136) (owner: 10Mholloway)
[17:11:16] <jdlrobson>	 paladox: https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#site.allowOriginRegex looks like what is needed
[17:12:04] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10Jdlrobson) Did something change regarding https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#site.allo...
[17:12:51] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10Jdlrobson) (and to be clear I'm only interested in read only requests here)
[17:13:01] <paladox>	 jdlrobson nope that's not it
[17:13:07] <paladox>	 it's due to phab's CSP policy
[17:17:21] <jdlrobson>	 did that change recently?
[17:17:21] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10Dzahn) I don't see allowOriginRegex in our Gerrit config at all. That should mean "By default, unset, denying all cross-ori...
[17:18:24] <paladox>	 jdlrobson appears to have
[17:18:35] <paladox>	 though i can't find where
[17:18:42] <paladox>	 it
[17:18:49] <paladox>	 *it's csp policy is:
[17:18:49] <paladox>	 content-security-policy: default-src https://phab.wmfusercontent.org; img-src https://phab.wmfusercontent.org data:; style-src https://phab.wmfusercontent.org 'unsafe-inline'; script-src https://phab.wmfusercontent.org; connect-src 'self'; frame-src 'self' https://commons.wikimedia.org; frame-ancestors 'none'; object-src 'none'; form-action 'self'; base-uri 'none'
[17:18:52] <paladox>	 jdlrobson ^^
[17:21:03] <wikibugs>	 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: No longer possible to make CORS requests from Phabricator to Gerrit - https://phabricator.wikimedia.org/T218308 (10Dzahn) It should be the CSP on the Phabricator side.
[17:23:05] <wikibugs>	 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants redundant "mixed|null" - https://phabricator.wikimedia.org/T218324 (10Anomie)
[17:23:37] <tgr>	 hi releng! is https://gerrit.wikimedia.org/r/admin/groups/3fdcf8fd0d569e90a3e9b39788a29f2c50d33be9,members supposed to be the same thing as https://github.com/wikimedia/puppet/blob/874d540ac575f0210be86d610a0720e8d66270e2/modules/admin/data/data.yaml#L64-L78 ?
[17:24:16] <tgr>	 mdholloway is not listed in the former, we are trying to figure out what needs to be done to fix that
[17:24:41] <tgr>	 (also why is Kaldari called Valerie in that list?)
[17:24:55] <paladox>	 because of gerrit problems last year tgr 
[17:25:17] <paladox>	 basically kaldari was locked out due to some gerrit error and so he created a new account until we fixed his original account.
[17:36:52] <tgr>	 I see
[17:37:32] <thcipriani>	 tgr: hrm, I have a vague memory that deployment was for control of something else besides deployment of mediawiki...keyholder or something. That is, one could be part of the latter to deploy any service, but only part of the former to deploy mediawiki. If that was the case, it doesn't seem to be the case now looking at the sudo rules.
[17:38:42] <thcipriani>	 tgr: at any rate, I can add mdholloway to the list of wmf_deployers in gerrit.
[17:38:50] <thcipriani>	 er wmf-deployment that is
[17:39:34] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10zeljkofilipin) a:05brennen→03zeljkofilipin
[17:39:39] <thcipriani>	 I'll have to dig deeper to figure out if they should actually match or if there was some other weird rename going on.
[17:40:10] <wikibugs>	 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10zeljkofilipin)
[17:40:12] <mdholloway>	 thcipriani: thanks! My gerrit username is mholloway and not mdholloway, btw (sorry)
[17:40:33] <thcipriani>	 mdholloway: I was just about to ask as autocomplete was failing me
[17:41:12] <thcipriani>	 mdholloway: tgr {{done}}
[17:41:21] <tgr>	 thanks!
[17:41:32] <mdholloway>	 thanks!
[17:41:42] <thcipriani>	 yw :)
[17:44:50] <tgr>	 after a little digging, it seems like he was added in https://phabricator.wikimedia.org/T109855 so maybe mediawiki deployers and service deployers were handled separately?
[17:45:28] <greg-g>	 tgr: re kaldar.i's name showing valerie, I see it listed once at https://gerrit.wikimedia.org/r/q/owner:rkaldari%2540wikimedia.org 
[17:45:46] <greg-g>	 smells like an odd gitconfig issue on their end?
[17:46:56] <wikibugs>	 10Scap, 10serviceops: Scap2 to use etcd for target servers - https://phabricator.wikimedia.org/T218328 (10jijiki) p:05Triage→03Normal
[17:47:35] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki)
[17:47:37] <wikibugs>	 10Scap, 10serviceops: Scap2 to use etcd for target servers - https://phabricator.wikimedia.org/T218328 (10jijiki)
[17:48:11] <thcipriani>	 greg-g: fwiw, I think paladox is right on that one: I think it was a result of the UserNameToLowerCase issue we've had with a few users.
[17:48:34] <greg-g>	 oh, I missed that
[17:48:44] <greg-g>	 thanks paladox :)
[17:48:49] <paladox>	 your welcome :)
[17:49:04] <paladox>	 i think there is a task for that one some where.
[17:49:33] <tgr>	 thcipriani: so, seems like we should go through the formal process at https://wikitech.wikimedia.org/wiki/Production_shell_access#Additional_permissions_for_existing_users ?
[17:51:01] <thcipriani>	 tgr: hrm, it's not a escalation of shell access, not sure. greg-g ^
[17:51:37] <tgr>	 true, on the shell level there's no difference
[17:56:45] <James_F>	 Do we run php70 in production (as opposed to 72)? We've dropped the php71 checks from the SWAT gate pipeline but we still have the php70 ones…
[18:01:32] <thcipriani>	 scap uses 7.2 fwiw
[18:02:39] <greg-g>	 thcipriani: tgr: yeah, service deployers don't get added to wmf-deployments in Gerrit because that is basically for wmf-config and similar rights. But feel free to add him though since he's already a service depoyer
[18:08:15] <apergos>	 7.2 everywhere I believe
[18:14:07] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10zeljkofilipin)
[18:14:54] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10zeljkofilipin) p:05Triage→03Normal
[18:15:27] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Tekton Pipeline - https://phabricator.wikimedia.org/T217912 (10zeljkofilipin) p:05Triage→03Normal
[18:17:52] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10zeljkofilipin)
[18:18:17] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10zeljkofilipin) p:05Triage→03Normal
[18:18:42] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Jenkins - https://phabricator.wikimedia.org/T218333 (10zeljkofilipin)
[18:18:50] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Jenkins - https://phabricator.wikimedia.org/T218333 (10zeljkofilipin) p:05Triage→03Normal
[18:18:59] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Jenkins X - https://phabricator.wikimedia.org/T218334 (10zeljkofilipin)
[18:19:58] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Jenkins X - https://phabricator.wikimedia.org/T218334 (10zeljkofilipin) p:05Triage→03Normal
[18:20:14] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Jenkins X - https://phabricator.wikimedia.org/T218334 (10zeljkofilipin) a:03dduvall
[18:20:41] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Spinnaker - https://phabricator.wikimedia.org/T218335 (10zeljkofilipin)
[18:20:53] <wikibugs>	 10Release-Engineering-Team (Kanban): Evaluate Spinnaker - https://phabricator.wikimedia.org/T218335 (10zeljkofilipin) p:05Triage→03Normal
[18:21:20] <wikibugs>	 (03PS1) 10Umherirrender: [Disambiguator] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496507
[18:25:00] <wikibugs>	 (03PS1) 10Umherirrender: [Listings] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496510
[18:29:34] <wikibugs>	 (03PS1) 10Umherirrender: [Insider] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496516
[18:35:18] <wikibugs>	 (03PS1) 10Umherirrender: [GeoCrumbs] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496520
[18:38:42] <wikibugs>	 (03PS1) 10Umherirrender: [Josa] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496524
[18:55:43] <hasharDinner>	 oh my god
[18:58:10] <hauskatze>	 Mon Dieu
[19:00:04] <paladox>	 Mon Dieu?
[19:01:01] <wikibugs>	 10Release-Engineering-Team, 10serviceops: Our docker base images lack tags - https://phabricator.wikimedia.org/T218342 (10hashar)
[19:01:06] <hashar>	 yeah
[19:01:07] <hashar>	 exactly
[19:26:22] <wikibugs>	 (03PS1) 10Kosta Harlan: GrowthExperiments: Add MobileFrontend and VisualEditor dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/496559 (https://phabricator.wikimedia.org/T218345)
[19:27:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Wikimedia-Logstash, 10Discovery-Search (Current work), 10Patch-For-Review: ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156 (10EBernhardson) This looks to be working again and ingesting apifeature...
[19:28:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] GrowthExperiments: Add MobileFrontend and VisualEditor dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/496559 (https://phabricator.wikimedia.org/T218345) (owner: 10Kosta Harlan)
[19:45:08] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10Jdforrester-WMF) >>! In T206675#5024498, @herron wrote: > There looks to be a significant increase (about 1.5 mil...
[19:46:34] <wikibugs>	 (03PS2) 10Kosta Harlan: GrowthExperiments: Add MobileFrontend and VisualEditor dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/496559 (https://phabricator.wikimedia.org/T218345)
[20:14:13] <wikibugs>	 (03PS1) 10Hashar: docker: ensure base images are up-to-date [integration/config] - 10https://gerrit.wikimedia.org/r/496579
[20:18:08] <wmf-insecte>	 Project beta-code-update-eqiad build #238765: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238765/
[20:20:38] <wmf-insecte>	 Project beta-update-databases-eqiad build #32423: 04FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32423/
[20:24:18] <wmf-insecte>	 Project beta-code-update-eqiad build #238766: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238766/
[20:28:23] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 11.36% of data above the critical threshold [3.0]
[20:28:31] <hashar>	 so
[20:28:41] <hashar>	 that is where I have no clue what is going to happen
[20:28:44] <hashar>	 rebuilding ci images
[20:28:47] <hashar>	 and hoping for the best ;((((
[20:29:47] <wikibugs>	 (03PS1) 10Hashar: docker: rebuild ci-stretch for debian/libc6 update [integration/config] - 10https://gerrit.wikimedia.org/r/496608 (https://phabricator.wikimedia.org/T216384)
[20:34:18] <wmf-insecte>	 Yippee, build fixed!
[20:34:19] <wmf-insecte>	 Project beta-code-update-eqiad build #238767: 09FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238767/
[20:35:08] <wikibugs>	 (03PS1) 10Jforrester: [WikimediaEditorTasks] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496610 (https://phabricator.wikimedia.org/T218136)
[20:36:23] <wikibugs>	 (03CR) 10Mholloway: [C: 03+1] [WikimediaEditorTasks] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496610 (https://phabricator.wikimedia.org/T218136) (owner: 10Jforrester)
[20:39:57] <wikibugs>	 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search, 10MW-1.32-release: Build failures for Elastica and CirrusSearch on REL1_32 - https://phabricator.wikimedia.org/T216612 (10EBernhardson)
[20:41:52] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble: Quibble space separated options shallow arguments - https://phabricator.wikimedia.org/T218357 (10hashar)
[20:41:54] <thcipriani>	 hashar: so re:gerrit deployment. I'd like to start running smoke tests for our setup in CI. paladox has some docker work to ensure that gerrit is able to build, but I'd like to stand it up and test it and make sure the plugins are all integrated properly (so we can avoid problems like we had with the healthcheck plugin changing last time I rolled forward)....is the "how to do that" in current ci
[20:41:56] <thcipriani>	 just another docker image?
[20:42:23] <paladox>	 :)
[20:43:31] <hashar>	 thcipriani: paladox I noticed some bazel/gerrit image for ci
[20:43:33] <hashar>	 but well
[20:43:38] <hashar>	 really
[20:43:48] <hashar>	 I have to raise a stackoverflow error for this week
[20:44:14] <hashar>	 I am at 40+hours already and don't even remember what I was supposed to work on early last week :'(
[20:44:58] <paladox>	 https://gerrit.wikimedia.org/r/#/c/integration/config/+/493638/ and https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/ is the changes
[20:45:28] <thcipriani>	 hashar: no worries. we're now at the latest version, they have yet to release a 2.15.12, but that's the version I'd like to target having actual integration tests running for. I'll ping you at some point before then to collect your thoughts.
[20:46:22] <wikibugs>	 (03PS2) 10Hashar: Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357)
[20:46:34] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Add Extension:WikimediaEditorTasks [tools/release] - 10https://gerrit.wikimedia.org/r/496484 (https://phabricator.wikimedia.org/T218136) (owner: 10Mholloway)
[20:47:19] <wikibugs>	 (03Merged) 10jenkins-bot: Add Extension:WikimediaEditorTasks [tools/release] - 10https://gerrit.wikimedia.org/r/496484 (https://phabricator.wikimedia.org/T218136) (owner: 10Mholloway)
[20:47:21] <paladox>	 2.15.12 may be released next week due to the severity of a bug discovered (gc)
[20:49:08] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "WIP see comments" (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) (owner: 10Hashar)
[20:54:50] <wikibugs>	 (03PS2) 10Hashar: docker: rebuild ci-stretch for debian/libc6 update [integration/config] - 10https://gerrit.wikimedia.org/r/496608 (https://phabricator.wikimedia.org/T216384)
[20:55:16] <hashar>	 thcipriani: so eventually:  /etc/hosts:208.80.154.85 gerrit.wikimedia.org
[20:56:05] <hashar>	 thcipriani: paladox for gerrit. yeah we will need a bazel container grabbing it from Google apt repo: http://storage.googleapis.com/bazel-apt stable/jdk1.8
[20:56:16] <paladox>	 yup
[20:56:18] <hashar>	 then from there I guess it is known business
[20:56:28] <hashar>	 clone with ci-src-setup
[20:56:42] <hashar>	 (set the env variable to process submodules, see the run.sh script)
[20:56:53] <hashar>	 then well invoke the releng/bazel container
[20:57:08] <hashar>	 if the clone of gerrit is too long
[20:57:19] <hashar>	 we can have it mirrored on the docker instances (that is defined somewhere in puppet)
[20:57:21] <paladox>	 gerrit will require a large timeout
[20:57:26] <hashar>	 so that we have a copy under /srv/git on all instances
[20:57:59] <hashar>	 and then    docker run --volume /srv/git:/srv/git:ro
[20:58:23] <hashar>	 and probably something like:  git clone --reference /srv/git/$ZUUL_PROJECT  $ZUUL_URL/$ZUUL_PROJECT
[20:58:32] <hashar>	 I don't think ci-src-setup supports that though
[20:58:50] <wikibugs>	 (03PS4) 10Paladox: Docker: Add bazel image [integration/config] - 10https://gerrit.wikimedia.org/r/493638
[20:58:53] <wikibugs>	 (03PS21) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328
[20:59:09] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] docker: rebuild ci-stretch for debian/libc6 update [integration/config] - 10https://gerrit.wikimedia.org/r/496608 (https://phabricator.wikimedia.org/T216384) (owner: 10Hashar)
[21:00:06] <hashar>	 ^^^
[21:00:09] <hashar>	 and really
[21:00:15] <hashar>	 we need to find a solution to that problem
[21:00:33] <hashar>	 magically upgrading everything is annoying
[21:00:41] <hashar>	 and the more I think about it
[21:00:51] <hashar>	 the more I think jobs should just point to :latest
[21:01:00] <wikibugs>	 (03Merged) 10jenkins-bot: docker: rebuild ci-stretch for debian/libc6 update [integration/config] - 10https://gerrit.wikimedia.org/r/496608 (https://phabricator.wikimedia.org/T216384) (owner: 10Hashar)
[21:01:04] <hashar>	 and we have a cron job that rebuild all the images on an hourly basis
[21:01:05] <paladox>	 hashar ci-src-setup uses jessie.
[21:01:14] <paladox>	 So does that mean we have to use jessie for the gerrit image?
[21:01:18] <hashar>	 paladox: it is fine. It is only there to git clone
[21:01:25] <paladox>	 ah ok
[21:01:57] <hashar>	 we will have to phase out jessie eventually
[21:02:06] <paladox>	 ok :)
[21:02:10] <hashar>	 but really I am fed up with migrations
[21:02:14] * paladox try's ci-src-setup
[21:05:24] <shinken-wm>	 PROBLEM - Host deployment-sessionstore01 is DOWN: CRITICAL - Host Unreachable (172.16.3.4)
[21:05:41] <wikibugs>	 (03PS1) 10Hashar: jjb: update quibble HHVM container for libc update [integration/config] - 10https://gerrit.wikimedia.org/r/496620 (https://phabricator.wikimedia.org/T216689)
[21:06:21] <paladox>	 hashar something like https://github.com/wikimedia/integration-config/blob/5247fb4649267a4105f5e3d705e0e0f60f5583ac/dockerfiles/jsduck/example-run.sh#L7 ?
[21:06:35] <paladox>	 so it warms the cache up, but still allows you to choose different branches?
[21:06:47] <paladox>	 well i mean it clones it
[21:06:57] <paladox>	 so it's all ready
[21:18:08] <wikibugs>	 10Continuous-Integration-Config, 10Discovery-Search (Current work), 10Patch-For-Review: Maven Wrapper does not support XDG_CACHE_HOME - https://phabricator.wikimedia.org/T218099 (10debt) 05Open→03Resolved a:03debt
[21:21:16] <wmf-insecte>	 Yippee, build fixed!
[21:21:16] <wmf-insecte>	 Project beta-update-databases-eqiad build #32424: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32424/
[21:21:21] <hashar>	 !log Updated quibble-vendor-mysql-hhvm-docker with latest libc6 hopefully fixing HHVM segfault within libpthread # T216689
[21:21:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:21:23] <stashbot>	 T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689
[21:23:48] <paladox>	 hashar hmm, im not sure how we use ci-src-setup (looking at other examples)
[21:24:01] <paladox>	 im trying to find run.sh that processes submodules
[21:25:32] <paladox>	 i've also noticed https://github.com/wikimedia/integration-config/blob/852e40e83be9aa07de346f09d19fb986298921cf/jjb/macro-docker.yaml#L207
[21:25:36] <paladox>	 that should remove the /p
[21:25:45] <hashar>	 !log Manually triggered tests for 12 ContentTranslation changes that had label:verified=-1 # T216689
[21:25:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:26:00] <hashar>	 ci-src-setup-simple/run.sh:if [ -z "${GIT_NO_SUBMODULES:-}" ]; then
[21:26:00] <hashar>	 ci-src-setup-simple/run.sh:    echo "\$GIT_NO_SUBMODULES set, skipping submodules"
[21:26:03] <hashar>	 paladox: ^^ sorry
[21:26:10] <paladox>	 ah
[21:26:11] <paladox>	 ok
[21:26:13] <hashar>	 it does submodules per default
[21:26:14] <paladox>	 thanks!!
[21:26:20] <hashar>	 the flag is to DISABLE processing submodules
[21:26:23] <hashar>	 so you can use it as is ;)
[21:26:28] <hashar>	 there is a macro in jjb for it
[21:26:47] <hashar>	 (all of that requires refactoring as well)
[21:26:55] <paladox>	 do we follow https://github.com/wikimedia/integration-config/blob/5247fb4649267a4105f5e3d705e0e0f60f5583ac/dockerfiles/npm-test-oojsui/example-run.sh#L9
[21:26:57] <hashar>	 that got done as a sprint/poc ages ago and never got polished
[21:27:01] <paladox>	 ok
[21:27:04] <paladox>	 ah
[21:27:10] <hashar>	 yeah that should work more or less ? :/
[21:27:19] <hashar>	 but that is the idea of ci-src-setup
[21:27:25] <hashar>	 use a container to clone
[21:27:28] <paladox>	 k
[21:27:40] <hashar>	 and the second container directly invoke the command but does not handle the cloning/source fetch etc
[21:27:50] <hashar>	 while other CI containers would do it 
[21:27:57] <hashar>	 typically quibble ones are autonomous
[21:28:00] <hashar>	 ideally
[21:28:17] <paladox>	 k
[21:28:25] <hashar>	 I would like to migrate to a model were a build only requires a single container that takes care of fetching source / installing and running the comand
[21:28:34] <hashar>	 but who knows what we will do with k8s ;)
[21:29:03] <paladox>	 heh
[21:29:14] <paladox>	 so i only need to copy https://github.com/wikimedia/integration-config/blob/5247fb4649267a4105f5e3d705e0e0f60f5583ac/dockerfiles/npm-test-oojsui/example-run.sh#L9 into run.sh on the gerrit image?
[21:30:00] <hashar>	 paladox: copy waht? 
[21:30:07] <paladox>	 hashar this https://github.com/wikimedia/integration-config/blob/5247fb4649267a4105f5e3d705e0e0f60f5583ac/dockerfiles/npm-test-oojsui/example-run.sh#L9
[21:30:17] <hashar>	 well that script is more of a test
[21:30:25] <paladox>	 oh
[21:30:28] <hashar>	 well an helper that one can manually run to verify the image is working ;)
[21:30:41] <hashar>	 which ideally should be run by CI when one propose a change to dockerfiles
[21:31:30] <paladox>	 hashar how do i use ci-setup? Or at least add gerrit to it :)?
[21:34:20] <hashar>	 paladox: on your machine?
[21:34:32] <hashar>	 you can replay the example-run.sh of npm-test-oojsui
[21:34:41] <hashar>	 and adjust the zuul_project zuul_xxxx variables
[21:34:42] <paladox>	 i guess but i mean to pre clone gerrit in the image
[21:34:51] <hashar>	 (and obviously the name of the second container)
[21:34:55] <hashar>	 to preclone 
[21:34:59] <hashar>	 well we dont do that
[21:35:03] <hashar>	 that make the container too large
[21:35:12] <hashar>	 the only exception has been operations-puppet 
[21:35:28] <hashar>	 for quibble jobs, whichc lone mediawiki and whatever combination of the thousands of extensions we have
[21:35:32] <hashar>	 we use a local cache
[21:35:49] <hashar>	 on the instance that run the docker container we have cloned a few high traffic repositories to /srv/git
[21:36:01] <paladox>	 ah, do you do that manually?
[21:36:03] <hashar>	 then we mount that iniside the container with --volume /srv/git:/srv/git:ro
[21:36:11] <hashar>	 so inside the container,  git can reference those
[21:36:21] <hashar>	 or clone from the /srv/git repo then fetch the patch and checkout
[21:36:43] <paladox>	 ah
[21:36:44] <hashar>	 but ci-src-setup-simple does not support that
[21:36:46] <paladox>	 so like:
[21:36:50] <paladox>	 https://github.com/wikimedia/integration-config/search?q=%2Fsrv%2Fgit%3Aro&unscoped_q=%2Fsrv%2Fgit%3Aro
[21:37:10] <hashar>	      - docker-log-dir
[21:37:10] <hashar>	      - docker-src-dir
[21:37:10] <hashar>	      - docker-ci-src-setup-simple
[21:37:10] <hashar>	      - docker-run-with-log-cache-src:
[21:37:10] <hashar>	          image: docker-registry.wikimedia.org/releng/jsduck:0.1.0
[21:37:11] <hashar>	          logdir: '/log'
[21:37:21] <hashar>	 (first job of jjb/job-templates.yaml )
[21:37:51] <hashar>	 the first two commands (with -dir suffixes) create directories on the host which are writable by anyone
[21:38:12] <hashar>	 docker-ci-src-setup-simple  invokes the ci-src-setup-simple container which populate the repository to ./src on the host
[21:38:15] <paladox>	 ok
[21:38:17] * paladox looks
[21:38:18] <hashar>	 and files belong to user nobody
[21:38:28] <paladox>	 gerrit will need a special user
[21:38:37] <hashar>	 then  docker-run-with-log-cache-src  is the magic macro that mounts ./src in the container and invoke the given image
[21:38:42] <paladox>	 bazel dislikes nobody && bower does not run under root
[21:38:49] <hashar>	 root : NO
[21:38:51] <hashar>	 it is vetoed
[21:39:02] <hashar>	 another user, to be investigated
[21:39:19] <hashar>	 we went with "nobody" because well it cant do much on the host in case something leaks out of the container
[21:39:20] <wikibugs>	 (03CR) 10Dduvall: [C: 04-1] "> Patch Set 2:" [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris)
[21:39:30] <paladox>	 https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/21/dockerfiles/gerrit/Dockerfile.template
[21:39:35] <paladox>	 i added "gerrit" as the user
[21:39:40] <hashar>	 we can check what is wrong with bazel when it is running as nobody though
[21:39:51] <hashar>	 for custom users really I don't know how it works
[21:39:53] <paladox>	 bazel fails to add the cache i think
[21:40:01] <hashar>	 it might need to exist on the host?
[21:40:15] <paladox>	 doing
[21:40:15] <paladox>	 # bower does not like root users thus needs a user account
[21:40:16] <paladox>	 RUN useradd gerrit -d /home/gerrit -m -s /bin/bash
[21:40:17] <paladox>	 worked
[21:40:27] <hashar>	 I remember I looked at using a custom user some ages ago. But cant remember off hand what was the conclusion
[21:40:30] <kaldari>	 greg-g: jdlrobson and/or the Editing team may need to ask for a Friday deployment to fix this editing bug on mobile: https://phabricator.wikimedia.org/T218352
[21:40:34] <hashar>	 beside that standardizing on nobody was a good thing
[21:40:41] <paladox>	 yup
[21:40:48] <hashar>	 userradd really I dont know what happens ;)
[21:40:50] <kaldari>	 if they aren't able to resolve today.
[21:41:05] <hashar>	 if you get some failures when running bazel as nobody, feel free to fill a task about it. I will be happy to investigate
[21:41:34] <hashar>	 we would also want to make sure bazel writes cacheable material (eg  stuff it download from the internet) to respect XDG_CACHE_CONFIG env variable
[21:41:34] <greg-g>	 kaldari: ack, I won't be here when you ask, most likely, so just Do The Right Thing, please :)
[21:41:39] <greg-g>	 (I'm taking most of the day off)
[21:41:44] <kaldari>	 got it. thanks!
[21:42:16] <hashar>	 paladox: else get Bazel to write cacheable things to $XDG_CACHE_CONFIG , or worth case scenario hardcode it to use /cache  (and not $HOME  since nobody does not have a home directory, it is set to /nonexistent which well ... does not exist)
[21:42:29] <hashar>	 AH
[21:42:32] <hashar>	 so folks
[21:42:34] <paladox>	 yeh
[21:42:35] <paladox>	 that's it
[21:42:38] <hashar>	 as much as I hate Docker
[21:42:45] <hashar>	 I tend to like zuul :)
[21:43:42] <hashar>	 but well
[21:44:04] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Release Pipeline, 10local-charts: Define a Blubberfile for mediawiki/core - https://phabricator.wikimedia.org/T218360 (10brennen)
[21:44:27] <hashar>	 paladox: anyway when starting from scratch, I usually just wrote a Dockerfile file  (not bothering with docker-pkg) 
[21:44:33] <hashar>	 then iterate until I have something working
[21:44:38] <paladox>	 ah ok
[21:44:42] <hashar>	 and writing an example-run.sh file
[21:44:43] <paladox>	 yeh the image i have works :)
[21:44:48] <hashar>	 but if you are fine with docker-pkg you can use it for sure
[21:44:54] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Release Pipeline, 10local-charts: Define a Blubberfile for mediawiki/core - https://phabricator.wikimedia.org/T218360 (10brennen)
[21:44:58] <wikibugs>	 10Release-Engineering-Team, 10Developer Productivity, 10local-charts, 10Epic: Create official docker images for Mediawiki and services used in the local development environment - https://phabricator.wikimedia.org/T217872 (10brennen)
[21:44:58] <paladox>	 the bazel image should be mostly fine
[21:45:08] <paladox>	 since it needs to install java / python && bazel
[21:45:15] <hashar>	 make sure to use docker-pkg from git and the latest version.  Then you can pass  --info to see the docker build output
[21:45:15] <paladox>	 * &&
[21:45:29] <hashar>	 and use --select='*bazel*'  to have it only build the bazel image
[21:45:32] <hashar>	 that is slightly faster
[21:45:47] <hashar>	 also the latest version process some stuff in parallel which make it faster (disclaimer: I have added that feature)
[21:50:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: integration-slave-jessie-1004 puppet error - https://phabricator.wikimedia.org/T218361 (10hashar)
[21:51:06] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Make it easier to update the query a Phabricator Dashboard "Query" Panel uses - https://phabricator.wikimedia.org/T113556 (10HappyDog) Sounds reasonable (I think).  One additional (automatic) step:  * Hide (delete/disable/remove ownership/whatever) the original query so it...
[21:52:56] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Make it easier to update the query a Phabricator Dashboard "Query" Panel uses - https://phabricator.wikimedia.org/T113556 (10epriestley) I'm not sure what you mean by that, can you show me a screenshot of what you mean by the "normal lists" where you're worried the query w...
[21:54:25] <wikibugs>	 (03PS22) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328
[21:54:30] <paladox>	 hashar like ^^?
[21:55:52] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10serviceops: Our docker base images lack tags - https://phabricator.wikimedia.org/T218342 (10greg)
[21:55:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: integration-slave-jessie-1004 puppet error - https://phabricator.wikimedia.org/T218361 (10hashar)
[21:56:09] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Script SSHFS setup in local-charts - https://phabricator.wikimedia.org/T218364 (10brennen)
[21:57:47] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Release Pipeline, 10local-charts: Define a Blubberfile for mediawiki/core - https://phabricator.wikimedia.org/T218360 (10brennen) p:05Triage→03Normal a:05brennen→03None
[21:58:09] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Make it easier to update the query a Phabricator Dashboard "Query" Panel uses - https://phabricator.wikimedia.org/T113556 (10HappyDog) Maybe I'm confusing things - when I use 'add existing panel' then all the old versions of the panel are still listed, currently.  But I do...
[21:59:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: integration-slave-jessie-1004 puppet error - https://phabricator.wikimedia.org/T218361 (10hashar) I do not know what is going. Seems to be an issue with the packages produced by sury.org.  We should probably just get rid of php on those integ...
[22:10:47] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add task status to phabricator notification mails - https://phabricator.wikimedia.org/T181001 (10hashar) Phabricator emails now have a `X-Phabricator-stamps` header which has a lot of useful informations. apparently implemented by upstream in February 2018 via https://secu...
[22:11:52] <jdlrobson>	 https://www.irccloud.com/pastebin/QTb22j3P/
[22:12:00] <jdlrobson>	 ^ is beta cluster down for others too? I'm getting  the above fatal
[22:13:13] <jdlrobson>	 opened https://phabricator.wikimedia.org/T218366
[22:13:30] <thcipriani>	 I can navigate to it it seems
[22:13:48] <thcipriani>	 https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page is 200, looks working anyway
[22:14:34] <thcipriani>	 ah, saw the anon users part
[22:15:12] <wikibugs>	 10Phabricator: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10hashar) @Jdrewniak I have seen someone complaining about reaching an url with `bug:T123 OR bug:T456`. I assume that came from Pherrit so I guess that one got solved?  Seems to be https...
[22:16:19] <jdlrobson>	 @thcipriani looks like it's account specific
[22:16:23] <jdlrobson>	 i tried a different account and it works fine
[22:16:53] <jdlrobson>	 I'm guessing it involves enabling 	Content Translation beta feature
[22:17:15] <thcipriani>	 beta feature would make sense
[22:19:37] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[22:23:23] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [2.0]
[22:23:39] <hashar>	 paladox: sorry gotta sleep ;(((
[22:23:44] <paladox>	 ok
[22:25:20] <wikibugs>	 (03CR) 10Krinkle: "Nice. Idea for later: a unit test for zuul layout to confirm all pipelines have a whitelist, or are for +2, or are for native git events (" [integration/config] - 10https://gerrit.wikimedia.org/r/493188 (https://phabricator.wikimedia.org/T192217) (owner: 10Hashar)
[22:25:49] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10dduvall) I'm pushing back on the pa...
[22:28:06] <Krenair>	 hmm
[22:28:12] <Krenair>	 the deployment blocker tasks
[22:28:37] <Krenair>	 do people start adding blockers there for things wrong on master that will make it into the next wmf branch
[22:28:46] <Krenair>	 or is that only done once the branching has taken place
[22:29:01] <Krenair>	 I feel like I should know the answer to this but
[22:29:09] <Krenair>	 am second guessing self
[22:30:02] <greg-g>	 Krenair: some do, yes
[22:30:09] <greg-g>	 and yeah, it's useful
[22:30:27] <greg-g>	 kinda a "make sure this is fixed or reverted before we roll out"
[22:31:17] <Krenair>	 boldly added one then
[22:35:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689
[22:38:10] <hashar>	 paladox: might try reviewing your change tomorrow. No promise though :/
[22:38:14] <hashar>	 sleep well everyone!
[22:38:17] <paladox>	 ok
[22:38:18] <paladox>	 thanks!
[22:42:54] <wikibugs>	 (03PS1) 10Krinkle: fresnel: Add --skip-deps arg to quibble invocation [integration/config] - 10https://gerrit.wikimedia.org/r/496681
[22:43:19] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] "Also, makes it even faster :)" [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:43:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] fresnel: Add --skip-deps arg to quibble invocation [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:43:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] fresnel: Add --skip-deps arg to quibble invocation [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:44:42] <wikibugs>	 (03PS2) 10Krinkle: fresnel: Add --skip-deps arg to quibble invocation [integration/config] - 10https://gerrit.wikimedia.org/r/496681
[22:44:48] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] "Updated." [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:46:03] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] "Saves about 45 second, for a 2min job. Yay" [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:47:18] <wikibugs>	 (03Merged) 10jenkins-bot: fresnel: Add --skip-deps arg to quibble invocation [integration/config] - 10https://gerrit.wikimedia.org/r/496681 (owner: 10Krinkle)
[22:49:28] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10mobrovac) How about having a semi-c...
[22:53:07] <wikibugs>	 10Release-Engineering-Team (Backlog), 10Release Pipeline (Blubber): Update blubber parser to v4 - https://phabricator.wikimedia.org/T218142 (10greg)
[22:53:13] <wikibugs>	 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10local-charts, 10Epic: Create official docker images for Mediawiki and services used in the local development environment - https://phabricator.wikimedia.org/T217872 (10greg)
[23:47:50] <wikibugs>	 (03PS1) 10Krinkle: Remove a few redundant mediawiki/job quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/496688
[23:48:37] <Krinkle>	 !log Abort job quibble-vendor-mysql-hhvm-docker/39874/ for mwext-CentralAuth (stuck after 59 minutes)
[23:48:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:55:07] <wikibugs>	 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: Omit "npm install" step in Fresnel job output - https://phabricator.wikimedia.org/T218374 (10Krinkle)
[23:58:22] <wikibugs>	 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: Omit "npm install" step in Fresnel job output - https://phabricator.wikimedia.org/T218374 (10Krinkle) Come to think of it, this step isn't needed at all.  Removed now with <https://gerrit.wikimedia.org/r/#/c/integration/config/+/496681> / {00a95...