[00:19:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675 (10bd808) @hashar you could try setting...
[05:36:45] <wikibugs>	 10Phabricator, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10ArielGlenn) Adding @mmodell since this is going to be something related to the phab puppet module or the phabricator-tools/public_task_dump.py script.
[07:21:50] <apergos>	 folks, I'm going to batch upload about 1k 32x32 pixel icons to beta commons, with 15 seconds between uploads. This shouldn't take much space (total space of images is ~ 4M), and shouldn't be a burden on the servers spaced out so slowly
[07:22:42] <apergos>	 I'll be adding captions to these later today, part of an initiative to get some images set up for a good strutured data testbed
[07:23:12] <apergos>	 those will also be with 15 seconds between requests
[07:23:21] <apergos>	 any issues, let me know
[07:30:48] <apergos>	 I'll likely do more batches of these in the coming days with different fonts or colors or whatever, and different captions
[07:35:23] <elukey>	 folks puppet on deployment-prep is still not syncing with prod
[07:35:53] <elukey>	 what is the procedure to fix the rebase stack to allow git-sync to work?
[07:36:16] <elukey>	 is it ok to remove the problematic cherrypick or do we usually wait the author?
[07:44:41] <apergos>	 if the cherry-pick is alredy merged then you can remove it; if it just needs to be manually rebased properly, you can do that
[07:44:55] <apergos>	 otherwise I would poke the author or 
[07:45:06] <apergos>	 I think Krenair (?) does a lot of the fixup work
[07:45:14] <apergos>	 elukey: 
[07:51:40] <elukey>	 apergos: thanks! the commit is marked as LOCAL HACK, not sure if it is already merged, I think that it clashes with a newer commit
[07:51:54] <elukey>	 I pinged Krenair but not response, he may be afk
[07:53:29] <apergos>	 wrong tx I think
[07:53:40] <apergos>	 the local hack ones are ones we want to keep as a general rule
[07:54:32] <apergos>	 s/tx/tz/
[08:10:16] <elukey>	 yeah I understand the use case, but it should be managed in a better way probably
[08:10:57] <elukey>	 I can work on other stuff in the meantime but I wanted to add a memcached shard to the mcrouter beta config :)
[08:16:47] <apergos>	 well it's been an ongoing pain point for us for years but there has never been a good workaround found
[08:21:05] <elukey>	 yep I can imagine
[09:45:14] <hashar>	 !log Purging Docker images on CI instances
[09:45:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:48:22] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, and 2 others: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10Daimona) >>! In T226945#5370566, @Legoktm wrote: > We absolutely need phan for release bra...
[09:53:51] <wikibugs>	 10Phabricator: Remove Phabricator 2FA for awight - https://phabricator.wikimedia.org/T236765 (10awight)
[10:02:09] <apergos>	 hm I hit an "upload throttle', well that puts a damper in my plans :-/
[10:03:20] <apergos>	 wonder how I can get an exemption for my account
[10:05:27] <apergos>	 I see, need to get added to 'autopatrolled'
[10:07:48] <apergos>	 guess I"ll move on to captioning these 300 for now and wait for the time to pass
[10:12:44] <dcausse>	 hi, are there dashboards/logstash instances running in deployment-prep where I could check if the jobqueue is running properly?
[10:14:41] <p858snake>	 apergos: which project? most have a confirmed group to override that
[10:14:51] <apergos>	 commons.beta
[10:14:59] <apergos>	 user:ArielGlenn
[10:15:10] <p858snake>	 oh beta
[10:15:27] <p858snake>	 bribe hashar or someone to make you a sysop would probably be easiest
[10:15:42] <p858snake>	 greg-g maybe
[10:19:17] <apergos>	 heh
[10:19:24] <apergos>	 I'll wait til someone's around in the sf tz
[10:20:17] <apergos>	 no worries, I have to fix up this version of the captioning code and then run it on the existing images with the same slow pace
[10:25:28] <hashar>	 p858snake: greg is in California so he is definitely sleeping 
[10:26:51] <hashar>	 dcausse: logstash is broken on beta, and has been broken since August at least :(  https://phabricator.wikimedia.org/T233134  """logstash-beta.wmflabs.org does not receive any mediawiki events"""
[10:27:31] <dcausse>	 hashar: by chance, do you know if the jobqueue is broken as well?
[10:28:11] <dcausse>	 I don't really know how to check that now without digging into kafka offsets
[10:28:12] <hashar>	 dcausse: I have no ide ahow the job queue runs nowadays :-\
[10:28:22] <hashar>	 something about EventBus / Kafka and whatever magic that got made 
[10:28:38] <dcausse>	 ok, thanks
[10:28:42] <hashar>	 but i don't even know whether beta has been upgraded to that new system as well :\
[10:28:48] <hashar>	 maybe mobrovac would know
[10:29:20] <hashar>	 apergos: what is your account name on beta?
[10:29:32] <apergos>	 (12:14:59 μμ) apergos: user:ArielGlenn
[10:29:42] <apergos>	 kinda boring, I know
[10:29:45] <hashar>	 :))
[10:29:46] <hashar>	 https://meta.wikimedia.beta.wmflabs.org/w/index.php?user=ArielGlenn&title=Special%3AGlobalUserRights
[10:29:51] <hashar>	 that is the global user rights
[10:29:55] <hashar>	 no clue which one to tick though
[10:30:00] <hashar>	 nor wether you get access to that page
[10:30:10] <apergos>	 I only want autopatrolled, nothing more
[10:30:54] <hashar>	 I gave you "global abuse filter manager", "global developer", "steward"   just like me
[10:30:59] <hashar>	 additionally granted "staff"
[10:31:09] <apergos>	 uh
[10:31:15] <apergos>	 ok but that's way more than I want/need
[10:31:21] <hashar>	 and "global autopatrolled"
[10:31:28] <apergos>	 global autopatrolled is really all I need
[10:31:38] <hashar>	 so tentatively if you had no access to the page above, now you should https://meta.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/ArielGlenn :D
[10:31:40] <apergos>	 what is 'global developer' anyways?
[10:31:46] <hashar>	 and being a steward you should be able to do stuff
[10:31:50] <hashar>	 I have no idea
[10:31:56] <apergos>	 yeah  I didn't see a list before, and now I do
[10:31:56] <hashar>	 that page only list the groups
[10:31:58] <apergos>	 so that works
[10:32:02] <hashar>	 which rights they grant is somewhere else :]
[10:32:17] <hashar>	 well they get listed on https://meta.wikimedia.beta.wmflabs.org/wiki/Special:ListGroupRights
[10:32:54] <apergos>	 see 'developer' in there? because I don't :-D
[10:33:01] <hashar>	 and there is https://meta.wikimedia.beta.wmflabs.org/w/index.php?title=Special%3AUserRights&user=ArielGlenn 
[10:33:06] <hashar>	 which I think is local to the wiki
[10:33:13] <hashar>	 but I don't think there is a way to list all rights for a given account
[10:33:19] <apergos>	 on meta I don't care
[10:33:23] <apergos>	 this is just for commons
[10:34:00] <hashar>	 ah
[10:34:07] <hashar>	 well the meta wiki is were one can set global rights
[10:34:15] <apergos>	 (beta commons of course)
[10:34:18] <apergos>	 right
[10:34:25] <mobrovac>	 dcausse: jobqueue in beta?
[10:34:49] <dcausse>	 mobrovac: yes, do you know if it's working and where I could find some logs about it?
[10:34:50] <hashar>	 apergos: and I would hope that by granting you "global autopatrolled" via meta, that grants you autopatroll on every beta wikis
[10:35:11] <mobrovac>	 dcausse: looking
[10:35:53] <dcausse>	 thanks!
[10:36:17] <apergos>	 no but I was able to add myself :-D
[10:36:31] <apergos>	 thank you very much!
[10:36:34] <icinga-wm>	 PROBLEM - DPKG on contint2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages https://wikitech.wikimedia.org/wiki/Monitoring/dpkg
[10:36:42] <apergos>	 now if I can figure out what I have wrong with unicode and these captions...
[10:39:36] <icinga-wm>	 RECOVERY - DPKG on contint2001 is OK: All packages OK https://wikitech.wikimedia.org/wiki/Monitoring/dpkg
[10:40:37] <apergos>	 nothing. I left off the 'File:' prefix for the title :-/   welp, that works now
[10:40:43] <mobrovac>	 dcausse: i restarted cpjobqueue, it seems it was stuck and unable to produce events, it appears to be ingesting messages fine now
[10:43:42] <dcausse>	 mobrovac: great, thank you!
[10:43:43] <wikibugs>	 10Phabricator: Remove Phabricator 2FA for awight - https://phabricator.wikimedia.org/T236765 (10Aklapper) @awight: See https://www.mediawiki.org/wiki/Phabricator/Help/Two-factor_Authentication_Resets for options.
[10:49:53] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `Error: validation failed` during `make deploy` - https://phabricator.wikimedia.org/T236471 (10zeljkofilipin) 05Open→03Resolved a:03zeljkofilipin Fixed! 🎉
[10:50:16] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `Error: validation failed` during `make deploy` - https://phabricator.wikimedia.org/T236471 (10zeljkofilipin) a:05zeljkofilipin→03jeena
[11:19:27] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Remove php-xdebug from releases1001 / releases2001 - https://phabricator.wikimedia.org/T236774 (10hashar)
[11:19:57] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) >>! In T234418#5594961, @MoritzMuehlenhoff wrote: > Where does this error show up, in...
[11:20:10] <addshore>	 o/ is there a grafana for the beta wiki data?
[11:20:23] <addshore>	 so i guess reading data from labmon1001.eqiad.wmnet ?
[11:24:13] <addshore>	 aaah yes grafana-labs.wikimedia.org
[11:25:51] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Remove php-xdebug from releases1001 / releases2001 - https://phabricator.wikimedia.org/T236774 (10hashar)
[11:32:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10jijiki) a:03jijiki
[11:33:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) a:05jijiki→03None
[11:43:00] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: Can not log in to mediawiki - https://phabricator.wikimedia.org/T236776 (10zeljkofilipin)
[11:43:13] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: Can not log in to mediawiki - https://phabricator.wikimedia.org/T236776 (10zeljkofilipin) p:05Triage→03Normal
[11:45:34] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: Can not log in to mediawiki - https://phabricator.wikimedia.org/T236776 (10zeljkofilipin)
[12:09:18] <wikibugs>	 10Project-Admins: Create project Wiktionary-fr - https://phabricator.wikimedia.org/T236467 (10Darkdadaah) 05duplicate→03Open
[12:14:13] <Urbanecm>	 !log deployment-prep: delete from user_properties where up_user=11942 and up_property like "%growth%" and up_property like "%suggested%";
[12:14:14] <wikibugs>	 10Project-Admins: Create project Wiktionary-fr - https://phabricator.wikimedia.org/T236467 (10Darkdadaah) I'm reopening this task per the above comment: this is not a duplicate of T101948 .  The current requested project is to be used by the French Wiktionary Community: in French, by and for the contributors to...
[12:14:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:32:11] <hauskater>	 hmm https://integration.wikimedia.org/zuul/
[12:32:16] <hauskater>	 status.json ?
[12:33:24] <hauskater>	 hashar: ^^
[12:35:13] <wikibugs>	 (03PS1) 10Hashar: operations/debs/prometheus-ipsec-exporter add debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/546920
[12:37:19] <wikibugs>	 (03PS1) 10Hashar: Add debian-glue to cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/546921
[12:37:32] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] operations/debs/prometheus-ipsec-exporter add debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/546920 (owner: 10Hashar)
[12:37:36] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add debian-glue to cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/546921 (owner: 10Hashar)
[12:40:12] <wikibugs>	 (03Merged) 10jenkins-bot: operations/debs/prometheus-ipsec-exporter add debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/546920 (owner: 10Hashar)
[12:40:14] <wikibugs>	 (03Merged) 10jenkins-bot: Add debian-glue to cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/546921 (owner: 10Hashar)
[12:45:58] <wikibugs>	 10Project-Admins: Create project Wiktionary-fr - https://phabricator.wikimedia.org/T236467 (10Aklapper) We are not going to have two different project tags for French Wiktionary that nobody will be able to distinguish... There can be a project tag and it can be used by both the community and developers and admins.
[12:46:09] <wikibugs>	 10Project-Admins: Create project Wiktionary-fr - https://phabricator.wikimedia.org/T236467 (10Aklapper)
[12:46:11] <wikibugs>	 10Project-Admins: Replace tracking bug T76447 by new project tag "Wiktionary-fr" - https://phabricator.wikimedia.org/T101948 (10Aklapper)
[12:47:13] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Zuul reloaded :)" [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar)
[12:47:56] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Testing on https://gerrit.wikimedia.org/r/#/c/operations/debs/prometheus-ipsec-exporter/+/546922/" [integration/config] - 10https://gerrit.wikimedia.org/r/546920 (owner: 10Hashar)
[12:48:21] <apergos>	 back to uploads now, since I did the captions for the first 300 images
[12:48:56] <wikibugs>	 (03PS1) 10Pwirth: layout, parameter_functions: Add BlueSpiceExportTables, BlueSpiceUEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923
[12:51:04] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout, parameter_functions: Add BlueSpiceExportTables, BlueSpiceUEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923 (owner: 10Pwirth)
[12:53:18] <wikibugs>	 (03PS2) 10Pwirth: layout, parameter_functions: Add BlueSpiceExportTables, BlueSpiceUEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923
[12:54:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout, parameter_functions: Add BlueSpiceExportTables, BlueSpiceUEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923 (owner: 10Pwirth)
[12:57:34] <wikibugs>	 (03PS3) 10Pwirth: layout, parameter_functions: Add ExportTables, UEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923
[12:59:49] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout, parameter_functions: Add ExportTables, UEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923 (owner: 10Pwirth)
[13:02:51] <wikibugs>	 (03PS4) 10Pwirth: layout, parameter_functions: Add ExportTables, UEModuleTable2Excel [integration/config] - 10https://gerrit.wikimedia.org/r/546923
[13:05:12] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10Simetrical) In addition to the advantages of type hints that have already been mentioned, I'll point out that...
[13:17:04] <wikibugs>	 (03CR) 1020after4: [V: 03+2 C: 03+2] Build the url field from server + url [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) (owner: 1020after4)
[13:17:46] <wikibugs>	 (03Abandoned) 1020after4: Edit Project Config [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/543171 (owner: 1020after4)
[13:32:27] <wikibugs>	 (03PS1) 10Hashar: Allow force push to upstream to populate repo [debs/prometheus-ipsec-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/546931
[13:32:58] <wikibugs>	 (03CR) 10Hashar: "That is to push master of https://github.com/dennisstritzke/ipsec_exporter to 'upstream' in Gerrit." [debs/prometheus-ipsec-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/546931 (owner: 10Hashar)
[13:33:03] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] Allow force push to upstream to populate repo [debs/prometheus-ipsec-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/546931 (owner: 10Hashar)
[13:33:13] <wikibugs>	 (03PS8) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248
[13:33:44] <andrewbogott>	 is gitiles the preferred way to browse gerrit repos?  Or are there better options?
[13:35:06] <wikibugs>	 (03CR) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4)
[13:37:19] <wikibugs>	 (03PS1) 10Hashar: Revert "Allow force push to upstream to populate repo" [debs/prometheus-ipsec-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/546932
[13:37:28] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Allow force push to upstream to populate repo" [debs/prometheus-ipsec-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/546932 (owner: 10Hashar)
[13:41:02] <apergos>	 that is certainly what I use
[13:45:03] <paladox>	 andrewbogott yes, or there's github.
[13:50:19] <awight>	 I see diffusion referenced in Phabricator tickets, but donno whether that's a best practice or not.
[13:51:01] <awight>	 :) we are so wiki to give 3 different answers
[13:53:24] <hashar>	 andrewbogott: Diffusion 
[13:53:46] <andrewbogott>	 diffusion only works for phab-hosted projects doesn't it?
[13:53:48] <hashar>	 gitiles  I think that is because that is well integrated with Gerrit
[13:53:59] <hashar>	 and we haven't managed to have Gerrit to point to Diffusion properly
[13:54:02] <hashar>	 na
[13:54:11] <hashar>	 all gerrit repositories are synced to Phabricator / Diffusion
[13:54:29] <andrewbogott>	 ok
[13:54:37] <hashar>	 and Phabricator thus process esall the commits which can then be referred to in task using the commit sha1
[13:54:39] <andrewbogott>	 so… lots of options, none of them preferred
[13:54:52] <paladox>	 hashar i thought gitiles was prefered?
[13:55:02] <hashar>	 I use Diffusion to point to commits in tasks
[13:55:11] <hashar>	 cause one just have to:  {sha1 of commit here}
[13:55:12] <paladox>	 As it easily integrates with gerrit, no sync delay
[13:55:33] <hashar>	 else I use gittiles from time to time, it is available from the Gerrit web ui and is super fast
[13:55:49] <hashar>	 github interface for git repo is sometime pleasant
[13:56:09] <andrewbogott>	 In my case I'm going to make a link that says something like "The history of this file can be viewed at <link>"
[13:56:13] <hashar>	 but really. I just git clone / git fetch / git show <treeish>:<file>|vim -   locally  most of the time
[13:56:17] <andrewbogott>	 which it sounds like gittiles is fine for
[13:57:03] <hashar>	 andrewbogott: +1 :]
[13:57:14] <paladox>	 yup
[13:57:16] <hashar>	 andrewbogott: and gittiles prefix is :  https://gerrit.wikimedia.org/g/
[13:57:27] <hashar>	 so eg  https://gerrit.wikimedia.org/g/mediawiki/core  should work
[13:57:44] <wikibugs>	 10Project-Admins: Replace tracking bug T76447 by new project tag "Wiktionary-fr" - https://phabricator.wikimedia.org/T101948 (10Darkdadaah) Ok so how about this: - The tag/tracking task in this initial request is pretty low priority for most people, and I don't think it is really useful in the end by itself: the...
[13:57:57] <andrewbogott>	 is that better than https://gerrit.wikimedia.org/r/plugins/gitiles/cloud ?
[13:58:09] <andrewbogott>	 I guess it's shorter :)
[13:58:34] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs failing with Composer TransportException: 404 Not Found (October... - https://phabricator.wikimedia.org/T236513
[13:59:40] <hashar>	 andrewbogott: yeah
[13:59:52] <hashar>	 sometime gittiles use the  /r/plugins/gitiles url
[13:59:55] <hashar>	 instead of just /g/
[14:00:33] <hashar>	 templates/apache.erb:    ProxyPass /g/ http://127.0.0.1:8080/r/plugins/gitiles/ retry=0 nocanon
[14:00:46] <paladox>	 gitiles defaults to /plugins/gitiles/ but we have a shortcut in the apache template
[14:01:10] <hashar>	 then in gitiles.config we have:
[14:01:13] <hashar>	 [gerrit]
[14:01:13] <apergos>	 I dslike surfing around in DIffusion so I don't point people there; gerrit is the source of truth so I like to send people to the tool it links to (gittiles)
[14:01:24] <hashar>	   baseUrl = https://${::gerrit::host}/g
[14:01:37] <hashar>	 paladox: so it seems gittiles does not always use the configured baseUrl :\
[14:01:50] <paladox>	 yeh
[14:05:12] <paladox>	 oh! Well it works in gerrit ui
[14:05:23] <paladox>	 just gitiles dosen't do it
[14:09:04] <paladox>	 hashar ^
[14:24:26] <hashar>	 !log integration-slave-jessie-1004 : updating buster cow image to verify the cron updater actually works: /usr/sbin/cowbuilder --update --basepath "/srv/pbuilder/base-buster-amd64.cow"
[14:24:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:27:51] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: cowbuilder cron update fails for buster - https://phabricator.wikimedia.org/T236796 (10hashar)
[14:31:06] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: cowbuilder cron update fails for buster - https://phabricator.wikimedia.org/T236796 (10hashar) ` # head /var/cache/pbuilder/base-buster-amd64.cow/var/lib/apt/lists/mirrors.wikimedia.org_debian_dists_buster_InRe...
[14:34:09] <hashar>	 !log integration-slave-jessie-* : rm /var/cache/pbuilder/base-buster-amd64.cow/var/lib/apt/lists/mirrors.wikimedia.org_debian_dists_buster_InRelease  && /usr/sbin/cowbuilder --update --basepath "/srv/pbuilder/base-buster-amd64.cow" # T236796
[14:34:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:34:12] <stashbot>	 T236796: cowbuilder cron update fails for buster - https://phabricator.wikimedia.org/T236796
[14:35:08] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: cowbuilder cron update fails for buster - https://phabricator.wikimedia.org/T236796 (10hashar) 05Open→03Resolved a:03hashar I have just moved the file: ` # mv /var/cache/pbuilder/base-buster-amd64.cow/var...
[14:52:07] <wikibugs>	 (03PS1) 10Urbanecm: Add Echo as phan dependency for CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/546955 (https://phabricator.wikimedia.org/T231395)
[15:02:01] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10puppet-compiler, 10Puppet: add compiler1003 to jenkins - https://phabricator.wikimedia.org/T236468 (10hashar) The instance has been added to Jenkins https://integration.wikimedia.org/ci/computer/compiler1003.puppet-diffs.eqiad.wmflabs/   Seems there are some foll...
[15:03:39] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10VisualEditor, and 6 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10hashar)
[15:32:02] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Remove php-xdebug from releases1001 / releases2001 - https://phabricator.wikimedia.org/T236774 (10Ottomata) a:03hashar Merged.  Ran `apt-get purge php-xdebug` on releases1001 an releases2...
[15:33:03] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10puppet-compiler, 10Puppet: add compiler1003 to jenkins - https://phabricator.wikimedia.org/T236468 (10jbond) @hashar i think if we get no more issue reports this week then we can rebuild 1001 and 1002 on Monday?
[15:33:26] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Remove php-xdebug from releases1001 / releases2001 - https://phabricator.wikimedia.org/T236774 (10Ottomata) p:05Triage→03Normal
[15:38:10] <wikibugs>	 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10kaldari) >I guess its not clear to me what exactly the decision to kill the server side...
[15:43:39] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Operations, 10Release Pipeline, 10serviceops, 10Goal: Self-service Deployment Pipeline - https://phabricator.wikimedia.org/T228676 (10Ottomata) a:03akosiaris @akosiaris, assigning to you, feel free to undo or reassign if this is not correct.
[16:11:24] <apergos>	 doing captions for the remaining ~700 icons now, and that's likely everything for today 
[16:28:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675 (10hashar) Ah thank you @bd808 though t...
[16:31:09] <hashar>	 !log integration-agent-docker-1005 change sudo and ldap client (to sssd) # T236675
[16:31:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:31:12] <stashbot>	 T236675: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675
[16:34:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675 (10hashar) I have switched `integration...
[16:49:19] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, and 2 others: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10Daimona) A quick note: in order to keep phan working on release branches, we cannot remove...
[16:51:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Technical-Debt: Local Phan errors that don't show up on CI - https://phabricator.wikimedia.org/T172935 (10Daimona) 05Open→03Resolved a:03WMDE-Fisch I don't see any actionable here. We're now on phan 1.3.2 for almost all repos, and some of t...
[17:00:29] <wikibugs>	 10Continuous-Integration-Config, 10phan-taint-check-plugin: phan-seccheck job should install dependencies via composer instead of removing extra things from mediawiki/vendor - https://phabricator.wikimedia.org/T188438 (10Daimona) Is this still a thing? Currently the job doesn't seem to be removing anything.
[17:08:39] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[17:09:06] <_joe_>	 thcipriani: ^^
[17:18:02] <greg-g>	 jenkins is doing things, but zuul status dashboard has a lot of empty changes?
[17:21:12] * thcipriani looks
[17:22:05] <wikibugs>	 (03PS1) 10Hashar: Handle fresnel non concurrency in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/546987 (https://phabricator.wikimedia.org/T234259)
[17:22:16] <thcipriani>	 looks like a patchset with a lot of dependent changes went through
[17:22:25] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, and 2 others: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10sbassett) >>! In T226945#5614208, @Daimona wrote: >>>! In T226945#5370566, @Legoktm wrote:...
[17:22:33] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "It is happening right now :-\" [integration/config] - 10https://gerrit.wikimedia.org/r/546987 (https://phabricator.wikimedia.org/T234259) (owner: 10Hashar)
[17:24:01] <thcipriani>	 also ^
[17:24:26] <hashar>	 so yeah 
[17:24:35] <hashar>	 someone has send bunch of patches to mediawiki/core like 5 or 6
[17:24:43] <hashar>	 each trigger multiple builds in multiple pipelines
[17:24:56] <hashar>	 and since that is a chain there is a large bump of  requests to craft merge commits
[17:25:00] <hashar>	 so it is really "just" a spike
[17:25:05] <hashar>	 we should tweak that alarm I guess
[17:25:23] <hashar>	 it seems too sensible
[17:25:46] <hashar>	 (+ there is that odd race between the jenkins build queue and the gearman plugin)
[17:25:50] <wikibugs>	 (03Merged) 10jenkins-bot: Handle fresnel non concurrency in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/546987 (https://phabricator.wikimedia.org/T234259) (owner: 10Hashar)
[17:26:04] <hashar>	 !log Reloading zuul for https://gerrit.wikimedia.org/r/546987
[17:26:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:28:25] <hashar>	 that unleashed all the pending https://integration.wikimedia.org/ci/job/mediawiki-fresnel-patch-docker/  builds in the Jenkins queue
[17:32:08] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Remove php-xdebug from releases1001 / releases2001 - https://phabricator.wikimedia.org/T236774 (10hashar) 05Open→03Resolved Awesome thank and it is indeed totally gone. Thank you for the manual cleanup!  Als...
[17:33:28] <wikibugs>	 (03PS1) 10Zfilipin: Capitalize MediaWiki user (from admin to Admin) [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546988
[17:34:08] <greg-g>	 hashar: s/sensible/sensitive/
[17:36:00] <hashar>	 greg-g: thank you :]
[17:36:16] <hashar>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/546989/ might tweak the probe
[17:36:32] <wikibugs>	 10Phabricator, 10Wikipedia-iOS-App-Backlog: Update Herald (H187) to include project 4245 - https://phabricator.wikimedia.org/T236821 (10LGoto)
[17:40:49] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[17:42:03] <apergos>	 so for anyone around (maybe greg-g, maybe someone else), is 15 seconds between uploads 9files are tiny: 32x32 png) overly cautious? can I cut it down to, I dunno, 10 seconds? Less?  
[17:42:31] <apergos>	 doing bulk uploads of icons and then adding captions (structured-data style) so we can have a bunch of data for sdc testing
[17:42:45] <apergos>	 15 seconds has apparently been fine, I've done that all day
[17:43:20] <wikibugs>	 (03PS1) 10Zfilipin: TLDR section in README [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546990
[17:44:25] <apergos>	 these are uploads to beta commons, forgot to say (it was mentioned much earlier in the backscroll, I announced when sf folks were sleeping)
[17:46:08] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10puppet-compiler, 10Puppet: add compiler1003 to jenkins - https://phabricator.wikimedia.org/T236468 (10hashar) Seems so yes :-]
[17:48:51] <wikibugs>	 (03CR) 10Brennen Bearnes: [C: 03+1] "Seems like a good idea to me.  Maybe could use a few comments describing what some of the commands do." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546990 (owner: 10Zfilipin)
[17:55:37] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin)
[18:14:33] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[18:19:03] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10Simetrical) Also, as we start using scalar type hints, we should require declare( strict_types = 1 ) for all f...
[18:21:18] <wikibugs>	 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Performance-Team (Radar): Perf regression from ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10WDoranWMF)
[18:22:31] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin)
[18:24:43] <wikibugs>	 10Release-Engineering-Team-TODO, 10Release Pipeline, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 (10WDoranWMF)
[18:28:50] <wikibugs>	 10Phabricator, 10Wikipedia-iOS-App-Backlog: Update Herald (H187) to include project 4245 - https://phabricator.wikimedia.org/T236821 (10MBinder_WMF) 05Open→03Resolved {meme, src="seal-of-approval"}
[18:40:32] <apergos>	 couple hundred captions to go and then done for the night
[18:45:03] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[18:47:35] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10Tgr) Strict types would result in lots of type errors around int/string conversions, for minimal gain. Not a u...
[18:49:52] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10Simetrical) What type of errors? Keeping in mind, again, that it only affects type hints on functions, not any...
[18:51:45] <andrewbogott>	 If anyone has a moment… I could use help understanding why I can't merge a patch.  Probably I'm doing something silly...
[18:51:46] <andrewbogott>	 https://gerrit.wikimedia.org/r/#/c/openstack/horizon/deploy/+/547000/
[18:52:19] <andrewbogott>	 I can't mark it as verified, but neither (as far as I know) is there a CI job which does that
[18:53:15] * mutante tries hitting the Verified button to see 
[18:53:32] <mutante>	 andrewbogott: for some reason i was able to V+2
[18:53:37] <mutante>	 do you have submit button now?
[18:53:47] <andrewbogott>	 yes!
[18:53:50] <andrewbogott>	 but… why can't I verify?
[18:54:09] <mutante>	 that's odd, usually everything is just based on LDAP ops
[18:54:15] <mutante>	 so we would be the same
[18:54:33] <andrewbogott>	 this definitely worked last time I worked on this repo
[18:54:56] <mutante>	 paladox: ^
[18:55:17] <mutante>	 hmm.. submodules are involved
[18:55:38] <paladox>	 !log granted ldap/ops ownership of https://gerrit.wikimedia.org/r/#/admin/projects/openstack/horizon/deploy,access
[18:55:40] <paladox>	 mutante ^
[18:55:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:56:05] <mutante>	 paladox: oh! that makes me wonder how i had that 
[18:56:24] <mutante>	 andrewbogott: fixed?
[18:56:39] <andrewbogott>	 yep, works now although I think paladox is still fine tuning
[18:56:39] <mutante>	 let me remove the V+2 again for testing
[18:57:08] <mutante>	 ok, already merged
[18:57:11] <andrewbogott>	 I merged already, sorry
[18:57:19] <mutante>	 np
[18:57:28] <mutante>	 paladox: thanks
[18:57:33] <paladox>	 yw :)
[18:57:39] <mutante>	 though i dont know why i was able to do that
[18:57:49] <mutante>	 it's not admin rights
[18:57:50] <andrewbogott>	 there will be another patch like that coming up once I rebuild the wheels
[18:57:58] <mutante>	 ok
[19:06:07] <paladox>	 !log updaed ownership of https://gerrit.wikimedia.org/r/#/admin/projects/openstack/horizon/deploy,access to wmcs-trusted
[19:06:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:22:47] <wikibugs>	 10Continuous-Integration-Config, 10Google-Code-in-2019: Add yourself to the Jenkins whitelist in Gerrit to trigger testing unit test failures or code style issues yourself - https://phabricator.wikimedia.org/T235286 (10Urbanecm) Imported into GCI site as https://codein.withgoogle.com/tasks/5191615979716610/
[20:00:04] <wmf-insecte>	 Project mwcore-phpunit-coverage-master build #266: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/266/
[20:08:09] <wikibugs>	 10Release-Engineering-Team-TODO, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Performance-Team (Radar): Perf regression from ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Krinkle)
[20:08:22] <wikibugs>	 10Release-Engineering-Team-TODO, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Performance-Team (Radar): Performance regression from ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Krinkle) p:05Normal→03High
[20:31:05] <wikibugs>	 (03PS1) 10Umherirrender: [DonationInterface] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/547031 (https://phabricator.wikimedia.org/T224763)
[20:47:03] <cdanis>	 is there a way to, in gerrit, view all unresolved comments on a given patch?  across all patchsets?
[20:51:54] <paladox>	 cdanis polygerrit from 2.16+
[20:52:18] <cdanis>	 aha, cool
[20:52:20] <cdanis>	 thanks paladox!
[20:52:26] <paladox>	 your welcome :)
[20:52:34] <paladox>	 cdanis the feature is called "change log"
[20:52:49] <paladox>	 well comment thread now
[20:57:26] <mutante>	 i guess we need a naming standard for keyholder keys. some are "service_deploy" some are "deploy_service" and some are just "service". and then of course they have to match in actually private repo and labs/private and so easy to make mistakes each time
[20:58:10] <paladox>	 oh
[20:58:15] <paladox>	 haha i confused change log
[20:58:29] <paladox>	 change log is the current view, comment thread is the new thing
[20:58:49] <mutante>	 i just made one for the deploy-design group but looked at existing keys for gerrit and phabricator, so it's called just "design". renaming it again 
[21:04:35] <greg-g>	 There's an AMA(Ask Me Anything)-like office hour in #wikimedia-tech for this year's Technical Conference. Join and ask away!
[21:07:32] <wikibugs>	 10Phabricator, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10mmodell) a:03mmodell >>! In T236507#5613861, @ArielGlenn wrote: > Adding @mmodell since this is going to be something related to the phab puppet module or the phabricator-tool...
[21:08:15] <wikibugs>	 10Phabricator, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10mmodell) p:05Triage→03Normal
[22:56:11] <wikibugs>	 10Phabricator, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10mmodell) So the schema didn't change as far as I can see. I can't actually find any reason why that part of the dump would be empty. I'll keep digging though.
[23:07:13] <wikibugs>	 10Phabricator, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10mmodell) Here is the relevant part of the dump code:  `lang=python  data['task'] = taskdata data['project'] = {} data['project']['projects'] = phabdb.get_projectbypolicy(pdb, po...