[00:00:39] <legoktm>	 Yeah
[00:00:52] <legoktm>	 And if it can stand-up to codesearch, I'll point extdist and libup at it too
[00:00:59] <James_F>	 Nice.
[00:02:16] <legoktm>	 it's powered by a 70 line Python script ^.^
[00:02:33] <legoktm>	 also, we need to clean up Gerrit
[00:03:03] <James_F>	 So very much, yes.
[00:04:01] <James_F>	 !log Docker: Complete rebuild finished and published; only took 2.5 hours. But now T228196 is fixed elsewise anyway, oh well.
[00:04:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[00:04:03] <stashbot>	 T228196: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196
[00:14:58] <Krinkle>	 legoktm: pull or push?
[00:15:07] <legoktm>	 pull
[00:15:23] <Krinkle>	 so codesearch used to have its own way of mirroring repos?
[00:15:48] <legoktm>	 it just clones them locally and pulls regularly
[00:15:56] <Krinkle>	 right
[00:16:05] <Krinkle>	 legoktm: and it still does, but from this mirror instead?
[00:16:14] <legoktm>	 yes
[00:16:17] <Krinkle>	 cool
[00:16:43] <legoktm>	 codesearch also kept like 3 or 4 copies of the same repo, so it would hit gerrit multiple times for the same stuff, and now it's getting deduplicated through the mirror
[00:16:53] <Krinkle>	 legoktm: what's the general latency people you'd expect from gerrit merge to indexed?  
[00:17:08] <Krinkle>	 s/people//
[00:17:46] <legoktm>	 the mirror is going to git fetch every hour, and I'm going to lower codesearch's pull frequency to every 30 minutes since we're not hitting Gerrit itself anymore, so 90 minutes give or take?
[00:18:18] <legoktm>	 (worst case)
[00:20:33] <James_F>	 How fast was it before?
[00:21:14] <legoktm>	 hourly iirc
[00:21:30] <legoktm>	 ..I have the code open, I don't know why I didn't just read it
[00:21:31] <legoktm>	 an hour
[00:22:20] <James_F>	 Right.
[00:26:54] <legoktm>	 Jul 17 00:26:31 codesearch4 docker[8610]: 2019/07/17 00:26:31 Failed to clone https://ggmirror.wmflabs.org/cgit/oojs/ui.git, see output below
[00:26:54] <legoktm>	 Jul 17 00:26:31 codesearch4 docker[8610]: Cloning into 'vcs-29f093d4447d83115130403a69195df9332e06d0'...
[00:26:54] <legoktm>	 Jul 17 00:26:31 codesearch4 docker[8610]: fatal: dumb http transport does not support shallow capabilities
[00:31:05] <legoktm>	 https://lists.zx2c4.com/pipermail/cgit/2015-January/002351.html sigh
[00:33:38] <James_F>	 legoktm: Fun.
[00:37:10] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: Provide ci-buster [integration/config] - 10https://gerrit.wikimedia.org/r/523833
[00:37:12] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: [lintr] Migrate from jessie all the way to buster [integration/config] - 10https://gerrit.wikimedia.org/r/523834
[00:37:14] <wikibugs>	 (03PS1) 10Jforrester: jjb: [lintr-docker] Migrate to buster-based image [integration/config] - 10https://gerrit.wikimedia.org/r/523835
[00:38:35] <James_F>	 Also, tah-dah, buster's here.
[00:40:14] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: Provide ci-buster [integration/config] - 10https://gerrit.wikimedia.org/r/523833 (owner: 10Jforrester)
[00:40:19] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: [lintr] Migrate from jessie all the way to buster [integration/config] - 10https://gerrit.wikimedia.org/r/523834 (owner: 10Jforrester)
[00:41:07] <James_F>	 (Mostly because a bunch of R packages haven't been back-ported from buster, but shhhhh. ;-))
[00:41:43] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: Provide ci-buster [integration/config] - 10https://gerrit.wikimedia.org/r/523833 (owner: 10Jforrester)
[00:41:46] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: [lintr] Migrate from jessie all the way to buster [integration/config] - 10https://gerrit.wikimedia.org/r/523834 (owner: 10Jforrester)
[00:42:17] <James_F>	 !log Docker: Deploy ci-buster and lintr:0.2.0
[00:42:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[00:58:29] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/523835 (owner: 10Jforrester)
[01:00:44] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [lintr-docker] Migrate to buster-based image [integration/config] - 10https://gerrit.wikimedia.org/r/523835 (owner: 10Jforrester)
[01:04:23] <legoktm>	 ok, mirror is working with smart http now
[01:04:36] <legoktm>	 turns out it's already built into git
[02:21:53] <legoktm>	 https://www.mediawiki.org/wiki/Gerrit/2019_cleanup
[03:09:09] <TimStarling>	 what does archiving mean on that page?
[03:12:50] <legoktm>	 TimStarling: usually, empty the repo leaving behind a README or ARCHIVED file, mark it as read-only in Gerrit, disable any CI, archive any phab projects, wiki pages, etc. We have a checklist for MW extensions/skins, but not a generic one yet
[03:13:35] <TimStarling>	 this is for security? or some other reason?
[03:13:56] <James_F>	 Reduce load on people doing security checks, things indexing repos, etc.
[03:14:09] <James_F>	 Avoid misleading people that this software might work or be supported.
[03:14:17] <legoktm>	 just cleanup, makes it obvious the repo isn't being used
[03:15:35] <legoktm>	 James_F: oh, I have the page sorted by date rather than repo
[03:17:21] <legoktm>	 do you want me to sort it the other way? I can just tweak the script
[03:19:06] <James_F>	 legoktm: Mostly I wanted to split out the operations/debs stuff from the rest, as it's mostly all of a type.
[03:19:40] <James_F>	 But yeah, alpha-sort is more likely for people to notice "their" old repos without having to read 2000 lines.
[03:19:56] <TimStarling>	 something like ircecho presumably works as well as ever, with no obvious security concerns, and it's hard to know if anyone is using it
[03:20:21] <TimStarling>	 making it so that "git pull" erases all the source files seems quite intrusive
[03:21:21] <legoktm>	 probably. though it'll need to get ported to Python 3 if people want to keep using it
[03:21:56] <James_F>	 Yeah, another argument in favour of blanking old code. But disruption for downstream users is a fair point.
[03:22:01] <TimStarling>	 will there be a documented way to get the code prior to archiving, aside from git checkout HEAD^ ?
[03:22:19] <legoktm>	 at least for MW, it's supposed to be intrusive so people know that the thing has been archived and no longer supported
[03:22:33] <legoktm>	 we could put that in the README
[03:22:57] <James_F>	 TimStarling: Example current practice is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Convert2Wiki/+/523811
[03:23:40] <James_F>	 TimStarling: I.e. you end up with a repo with just https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Convert2Wiki/+/master/ARCHIVED or similar in it.
[03:24:41] <James_F>	 Oh dear. https://www.mediawiki.org/wiki/Special:Watchlist -> "Internal error" for me.
[03:24:44] <TimStarling>	 for extensions, the security concerns are somewhat different than for a standalone thing that will work essentially forever if you can find its dependencies
[03:24:53] <legoktm>	 I think what we do will depend upon what exactly is in the repo. e.g. if it's just a fork or copy of some upstream (lots of ops stuff), we can blank it and leave a pointer to the upstream
[03:25:17] <legoktm>	 if it's custom code that's useful outside of Wikimedia, we should deliberate a bit more on whether blanking is best, or whether we should just mark it as unmaintained
[03:25:30] <James_F>	 Internal error and no <!-- parser report.
[03:25:36] <James_F>	 That's... a first for me, I think.
[03:25:46] <legoktm>	 wfm
[03:26:34] <TimStarling>	 what sort of error was it?
[03:26:37] <James_F>	 Specifically the firstHeading is <internalerror> and that's the end of the output. No stacktrace.
[03:27:04] <James_F>	 Aha. <div class="errorbox mw-content-ltr">[XS6VVQpAICkAACKDqjUAAABS] 2019-07-17 03:26:13: (internalerror-fatal-exception: Error, XS6VVQpAICkAACKDqjUAAABS, /wiki/Special:Watchlist?uselang=qqx)</div>
[03:27:14] <TimStarling>	 2019-07-17 03:24:24 [XS6U6ApAMFgAAFx6VfsAAABL] mw1253 mediawikiwiki 1.34.0-wmf.14 exception ERROR: [XS6U6ApAMFgAAFx6VfsAAABL] /wiki/Special:Watchlist   Error from line 197 of /srv/mediawiki/php-1.34.0-wmf.14/extensions/Flow/includes/Formatter/ChangesListFormatter.php: Call to a member function getAlphadecimal() on null {"exception_id":"XS6U6ApAMFgAAFx6VfsAAABL","exception_url":"/wiki/Special:Watchlist","caught_by":"mwe_handler"}
[03:27:26] <James_F>	 Fun.
[03:27:43] <TimStarling>	 your one is the same
[03:27:53] <James_F>	 Yes.
[03:29:07] <James_F>	 Looks like it's from StructuredDiscussions. I blame RoanKattouw.
[03:29:32] <RoanKattouw>	 I'll take a look
[03:30:08] <James_F>	 Possibly https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/522223 ?
[03:44:03] <RoanKattouw>	 No I don't think that's related
[03:45:09] <RoanKattouw>	 The code surrounding the code that threw the error looks very wrong
[03:45:13] <RoanKattouw>	 I'll investigate later
[03:46:08] <James_F>	 Thanks.
[03:57:30] <RoanKattouw>	 I'm confused why you even see this error, because it's using RecoverableErrorHandler. In theory it should log the error and move on
[04:01:41] <RoanKattouw>	 James_F: Could you please try to reproduce that error on mwdebug1002? I've got some code there that will log some more information
[04:03:10] <RoanKattouw>	 Also once you do that, I'm going to try reverting the commit you identified on mwdebug1002 and see if that fixes it. I don't see how it could cause it, but you're right that it's the only remotely related change in Flow in wmf.14
[05:59:23] <wikibugs>	 10Phabricator, 10DBA, 10Operations, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui)
[06:01:06] <wikibugs>	 10Phabricator, 10DBA, 10Operations, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui) p:05Triage→03Normal
[06:43:49] <wikibugs>	 10Release-Engineering-Team-TODO, 10MediaWiki-Debug-Logger, 10Operations, 10Wikimedia-Logstash: Logstash no longer captures DB queries in debug mode - https://phabricator.wikimedia.org/T190455 (10greg)
[06:44:47] <wikibugs>	 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO: jenkins-bot LDAP entry contains pmtpa references - https://phabricator.wikimedia.org/T198271 (10greg)
[06:46:25] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Puppet, 10User-greg: Preload TestingAccessWrapper in production mwrepl - https://phabricator.wikimedia.org/T143607 (10greg) 05Open→03Resolved a:03greg >>! In T143607#3413032, @EBernhardson wrote: > mwrepl has a 'bypa...
[06:59:18] <wikibugs>	 10Release-Engineering-Team-TODO, 10DBA, 10Reading-Infrastructure-Team-Backlog: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10Marostegui)
[06:59:48] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Technical-Debt: Investigate whether GD is still needed on appservers - https://phabricator.wikimedia.org/T227734 (10Legoktm) During the initial PHP 7 preparation (when that puppet file was written), I did an...
[07:02:02] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Technical-Debt: Investigate whether GD is still needed on appservers - https://phabricator.wikimedia.org/T227734 (10MoritzMuehlenhoff) On the Debian packaging level there are also no reverse depencies on php-...
[07:09:16] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Quibble, 10Documentation: Document how to add a new development dependency for an extension in Quibble - https://phabricator.wikimedia.org/T227909 (10hashar) Erik Bernhardson has hit the same issue to add `symfo...
[07:09:28] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Librarization, 10Quibble: Install extension require-dev dependencies in wmf-quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T220723 (10hashar)
[07:09:30] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Quibble, 10Documentation: Document how to add a new development dependency for an extension in Quibble - https://phabricator.wikimedia.org/T227909 (10hashar)
[07:13:12] <wikibugs>	 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10MoritzMuehlenhoff) Graphoid is based on NodeJS, so it should be migrated to Node 10 (an...
[07:13:53] <wikibugs>	 10Gerrit, 10Upstream: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10hashar) p:05Triage→03Normal
[07:35:15] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] [Convert2Wiki] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/523818 (https://phabricator.wikimedia.org/T228198) (owner: 10MarcoAurelio)
[07:36:52] <wikibugs>	 (03Merged) 10jenkins-bot: [Convert2Wiki] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/523818 (https://phabricator.wikimedia.org/T228198) (owner: 10MarcoAurelio)
[08:35:29] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[08:39:30] <wikibugs>	 10Phabricator, 10DBA, 10Operations, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui) Window reserved on the deployments page: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1832674&oldid=1832612 Em...
[08:44:08] <wikibugs>	 10MediaWiki-Releasing, 10Patch-For-Review: Review 1.33.1 tarball - https://phabricator.wikimedia.org/T227599 (10Legoktm) The full diffoscope output was too much to upload, so I only pasted the problematic part: {P8757}. I don't understand why there are changes to vendor/ when there are none in git: https://ger...
[08:46:22] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Technical-Debt: Investigate whether GD is still needed on appservers - https://phabricator.wikimedia.org/T227734 (10TheDJ) Note that getimagesize and getimagesizefromstring are [[ https://github.com/php/php-s...
[08:51:45] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:27:28] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "It is deployed now." [integration/config] - 10https://gerrit.wikimedia.org/r/523818 (https://phabricator.wikimedia.org/T228198) (owner: 10MarcoAurelio)
[09:48:25] <hashar>	 !log Reloaded Zuul [Convert2Wiki] Archive extension
[09:48:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:50:03] <hashar>	 Apple iPhone Xs : 1469€ vat included ....  That sounds hmm expensive! 
[09:51:03] <awight>	 Future doorstops and landfill bulk
[09:52:41] <hashar>	 yeah kind of :]
[09:52:46] <hashar>	 I need a new phone :\
[09:53:11] <hashar>	 had mine for four years and it is just terrible at getting the cells signal
[10:04:54] <wikibugs>	 (03PS1) 10Daimona Eaytoy: Add phan dependencies for PageForms [integration/config] - 10https://gerrit.wikimedia.org/r/523894 (https://phabricator.wikimedia.org/T228155)
[10:20:52] <p858snake>	 i still need to get a can of compressed air and try to fix my phone =/
[10:21:32] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip...
[10:55:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging: Investigate if puppetdbquery::query_resources should work on PCC - https://phabricator.wikimedia.org/T228266 (10jbond) p:05Triage→03Normal
[11:00:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging: Investigate if puppetdbquery::query_resources should work on PCC - https://phabricator.wikimedia.org/T228266 (10jbond) during my research i noticed that puppet db was failing with the following error  ` Compiling catalog for achernar.wikimed...
[11:02:48] <paladox>	 Hashar: that is expensive! :) (why is it €1469?) /me would have thought it would be €999
[11:03:13] <hauskatze>	 ,99 paladox , don't forget the ,99 :P
[11:03:20] <hauskatze>	 999,99.-
[11:03:28] <hauskatze>	 silly marketing :)
[11:03:53] <paladox>	 Oh
[11:04:13] <paladox>	 The euro is wierd :P in pounds that’s £999.99
[11:06:17] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: Investigate if puppetdbquery::query_resources should work on PCC - https://phabricator.wikimedia.org/T228266 (10jbond)
[11:10:12] <wikibugs>	 10Phabricator (Upstream), 10Regression, 10Upstream: In the "Merge Duplicates" dialog, clicking on a task title closes the dialog instead of selecting the task - https://phabricator.wikimedia.org/T228224 (10Aklapper) Can reproduce in current upstream.
[11:16:05] <wikibugs>	 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10akosiaris) >>! In T211881#5339132, @greg wrote: >>>! In T211881#5332195, @akosiaris wro...
[11:36:27] <wikibugs>	 (03CR) 10Michael Große: Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight)
[11:38:35] <wikibugs>	 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops, 10Patch-For-Review, 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10fsero) it seems that  container synchronization is broken a...
[11:39:42] <wikibugs>	 (03CR) 10Awight: [C: 04-1] Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight)
[11:41:51] <wikibugs>	 (03CR) 10Awight: [C: 03+1] "Thanks for adding this guard." [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan)
[11:53:27] <wikibugs>	 (03PS12) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116)
[11:56:29] <wikibugs>	 (03PS13) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116)
[12:00:31] <thedj>	 iphone X'es are quite expensive indeed. But as someoone who has owned lots of iphones... They tend to last a lot longer these days. I've gone from a yearly update cycle to a 3 year cycle.
[12:06:26] <thedj>	 i've owned several android devices as well btw. all made me cry within a year.
[12:09:06] <wikibugs>	 (03PS14) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116)
[12:09:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight)
[12:15:10] <wikibugs>	 (03PS15) 10Awight: Run node browser tests in each repo [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116)
[12:34:53] <paladox>	 thedj i've gone through around 3 iphones since 2014.
[12:35:12] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10TheDJ) {T228250} could be a potential blocker ?
[12:37:51] <thedj>	 i don't keep score exactly, but it's something like iphone 4, 4s, 5s, 6s and X
[12:38:27] <thedj>	 but my dad still uses my 4s, my 5s was bought by a friend of mine, and the 6s was sold to my mother
[12:38:50] <thedj>	 and we used to have tons of them for work, but not doing the app development thing anymore, so..
[12:39:03] <paladox>	 thedj i've gone through iPhone 5s, iPhone 6 plus && the iPhone X (which is my current)
[12:45:19] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) **Status**  There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.or...
[12:50:59] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10hashar) >>! In T220739#5340812, @TheDJ wrote: > {T228250} could be a potential blocker ?  That is for...
[12:56:35] <hashar>	 paladox: hi! in case you are still around here, does Gerrit 2.16 still requires a database backend ?
[12:57:07] <paladox>	 hashar technically yes. But we can switch it to use H2 (local db).
[12:57:15] <paladox>	 I've got that running on gerrit.git.wmflabs.org
[12:57:24] <paladox>	 and another larger gerrit install does too (gerrithub)
[12:57:39] <paladox>	 https://gerrit.git.wmflabs.org/r/ <-- is no longer running mysql :)
[12:58:24] <hashar>	 I thought about creating a Docker container to run Gerrit on my local machine :]
[12:59:07] <paladox>	 ahh, heh'
[13:23:31] <EJ62385>	 what up
[14:00:21] <wikibugs>	 (03PS8) 10Thcipriani: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma)
[14:05:26] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10LarsWirzenius)
[14:51:05] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Request access to deployment cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223698 (10akosiaris) 05Open→03Resolved a:03akosiaris User has been ad...
[14:55:02] <wikibugs>	 10Phabricator: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10epriestley) One of these is detecting as MIME type `video/ogg`; one as `application/ogg`. The `files.video-mime-types` configuration option in Phabricator includes both of these MIME types by default, bu...
[15:10:00] <icinga-wm>	 PROBLEM - DPKG on contint2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link
[15:18:24] <wikibugs>	 10Phabricator: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10Aklapper) `files.video-mime-types` is `{"video/mp4": true, "video/ogg": true, "video/webm": true, "video/quicktime": true, "application/ogg": true}` on our installation. We don't seem to change any setti...
[15:18:58] <icinga-wm>	 PROBLEM - jenkins_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins
[15:22:08] <icinga-wm>	 RECOVERY - jenkins_service_running on contint1001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins
[15:23:20] <James_F>	 RoanKattouw: Re-testing, now I get a 500 with "PHP fatal error: Nesting level too deep - recursive dependency?" My work wishlist works fine, though.
[15:23:58] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: Investigate if puppetdbquery::query_resources should work on PCC - https://phabricator.wikimedia.org/T228266 (10jbond)
[15:25:00] <icinga-wm>	 PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link
[15:26:33] <wikibugs>	 10Phabricator: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10matmarex) In that file, files.viewable-mime-types has application/ogg but not video/ogg. Maybe that’s it?
[15:31:12] <icinga-wm>	 ACKNOWLEDGEMENT - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 8 minutes ago with 2 failures. Failed resources (up to 3 shown): amusso apt broken due to python upgrade which triggers a replacement of zuul embedded python https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link
[15:32:12] <icinga-wm>	 PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[jenkins],Package[zuul] https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link
[15:33:57] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10Jdforrester-WMF)
[15:35:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Wikidata, and 3 others: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818 (10hashar) a:03hashar That job got copied from the one that builds wikim...
[15:36:14] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Reading-Infrastructure-Team-Backlog: Implement CI rules for new kartotherian repo - https://phabricator.wikimedia.org/T228170 (10Mholloway) >>! In T228170#5338071, @Jdforrester-WMF wrote: > OK, our bit here is don...
[15:36:25] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Reading-Infrastructure-Team-Backlog: Implement CI rules for new kartotherian repo - https://phabricator.wikimedia.org/T228170 (10LGoto) @MSantos is this done? Feel free to close if so, or move on the board as need...
[15:39:35] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907): Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hashar) a:03hashar
[15:39:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: PCC always has an ERROR when compiling for servers with  profile::redis::slave - https://phabricator.wikimedia.org/T228266 (10jbond)
[15:42:50] <icinga-wm>	 PROBLEM - DPKG on contint1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link
[15:43:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Technical-Debt: Cleanup CI puppet manifests - https://phabricator.wikimedia.org/T225735 (10hashar) p:05Triage→03Normal a:03hashar
[15:43:59] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Technical-Debt: Cleanup CI puppet manifests - https://phabricator.wikimedia.org/T225735 (10greg)
[15:44:16] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10Krinkle)
[15:44:37] <Krinkle>	 greg-g: liw: I think https://phabricator.wikimedia.org/T228292 is worth reverting for, at least commons.
[15:45:20] <liw>	 Krinkle, that's one of the ones I was going to ask about
[15:45:40] <liw>	 I'm gone for a couple of minutes, urgent break
[15:46:34] <Krinkle>	 liw: cool, good instinct :)
[15:48:16] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: PCC always has an ERROR when compiling for servers with  profile::redis::slave - https://phabricator.wikimedia.org/T228266 (10jbond) Investigating further this is due to how `populate_puppetdb` adds entries to the datab...
[15:51:33] <liw>	 I'm back
[15:53:05] <liw>	 Krinkle, should T228292 be UBN?
[15:53:06] <stashbot>	 T228292: API uploads fatal with UploadChunkFileException: Error storing file in '/tmp' backend-fail-internal - https://phabricator.wikimedia.org/T228292
[15:53:19] <Krinkle>	 liw: k, will ping on hangouts to resume call if ok?
[15:53:39] <liw>	 Krinkle, sure
[15:54:07] <Krinkle>	 yeah, train blockers always UBN. Sorry, I keep forgetting to do that :)
[15:54:21] <wikibugs>	 10Phabricator, 10Patch-For-Review: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10epriestley) Ah, good catch. Yeah, `files.viewable-mime-types` is likely the culprit here.
[15:54:59] <liw>	 Krinkle, I heard a ping but I don't know what to do find the cause
[15:55:31] <Krinkle>	 liw: https://hangouts.google.com/ (or if enabled, in gmail)
[15:56:24] <wikibugs>	 (03PS1) 10Hashar: jjb: inline npm-install macro to sole job using it [integration/config] - 10https://gerrit.wikimedia.org/r/523953 (https://phabricator.wikimedia.org/T227818)
[15:56:26] <wikibugs>	 (03PS1) 10Hashar: WDQS GUI: stop using legacy npm-setup.sh script [integration/config] - 10https://gerrit.wikimedia.org/r/523954 (https://phabricator.wikimedia.org/T227818)
[15:57:19] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) Thanks for setting that up for your tools, @Legoktm !  I think we probably still want a mirr...
[15:57:29] <wikibugs>	 (03PS4) 10Thcipriani: Add tests for scap.cli [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[15:57:32] <wikibugs>	 (03PS5) 10Thcipriani: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[15:58:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add tests for scap.cli [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[15:59:01] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[16:00:28] <wikibugs>	 (03CR) 10Jforrester: "Nice." [integration/config] - 10https://gerrit.wikimedia.org/r/523953 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:02:25] <thcipriani>	 hashar: was there a jenkins update in the last hour? trying to figure out what happened with https://integration.wikimedia.org/ci/job/service-pipeline-test/2497/console ?
[16:02:38] <hashar>	 thcipriani: yeah *LTS upgrade
[16:03:04] <hashar>	 https://jenkins.io/security/advisory/2019-07-17/
[16:03:04] <thcipriani>	 hrm, that method is provided by a plugin https://jenkins.io/doc/pipeline/steps/http_request/
[16:03:29] <hashar>	 there is something about CSRF protection tokens did not expire
[16:03:59] <hashar>	 hmm
[16:04:35] <hashar>	 and of course I forgot to read the changelog https://jenkins.io/changelog-stable/  :/
[16:04:55] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] WDQS GUI: stop using legacy npm-setup.sh script [integration/config] - 10https://gerrit.wikimedia.org/r/523954 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:05:11] <hashar>	 SMalyshev: going to update that wdqs gui build job
[16:05:28] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] jjb: inline npm-install macro to sole job using it [integration/config] - 10https://gerrit.wikimedia.org/r/523953 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:05:44] <hashar>	 thcipriani: maybe we can find out a minimal reproduction case ?   I thought about a pipeline job that just does the httpRequest method call
[16:06:12] <James_F>	 hashar: BTW, did you see I made ci-buster last night? Funt imes.
[16:06:17] <thcipriani>	 hashar: sure, we could do that, it looks like the plugin is not installed afaict...
[16:06:27] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "integration-config-jjb-diff-docker shows it is indeed a noop :]" [integration/config] - 10https://gerrit.wikimedia.org/r/523953 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:07:09] <thcipriani>	 but I'm unclear how that happened
[16:08:17] <hashar>	 !log Jenkins: updating wikidata-query-gui-build to get rid of slave-scripts/bin/npm-setup.sh which requires an obsolete/non existing tmpfs  # T227818
[16:08:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:08:19] <stashbot>	 T227818: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818
[16:09:01] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "wikidata-query-gui-build is the only job affected \o/" [integration/config] - 10https://gerrit.wikimedia.org/r/523954 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:09:05] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: inline npm-install macro to sole job using it [integration/config] - 10https://gerrit.wikimedia.org/r/523953 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:09:18] <wikibugs>	 10Project-Admins: Desktop Tag - https://phabricator.wikimedia.org/T225799 (10Aklapper) 05Open→03Resolved a:03Aklapper @JTannerWMF: Thanks a lot for the explanation! I rephrased the #desktop project tag description from > This tag is to identify what items should be built for Desktop platforms to > For item...
[16:09:37] <thcipriani>	 hashar: is if fine if I reinstall that plugin and restart if it wasn't meant to be uninstalled?
[16:10:22] <hashar>	 thcipriani: YES
[16:10:27] * thcipriani does
[16:10:46] <hashar>	 maybe I uninstalled it while upgrading some other plugins earlier this week / last week (they did some plugins security update)
[16:11:17] <hashar>	 thcipriani: oh also when uninstalling plugins, I think it just rename the .jar file
[16:11:46] <hashar>	 the plugins are under /var/lib/jenkins/plugins/  and are named  *.jpi
[16:11:49] <wikibugs>	 (03Merged) 10jenkins-bot: WDQS GUI: stop using legacy npm-setup.sh script [integration/config] - 10https://gerrit.wikimedia.org/r/523954 (https://phabricator.wikimedia.org/T227818) (owner: 10Hashar)
[16:11:57] <hashar>	 translation.jpi.disabled
[16:12:01] <hashar>	 that one I disabled it
[16:12:18] <thcipriani>	 the pipeline uses it to generate blubber files from blubberoid, so pipeline jobs would have been failing, but they were working this morning, so it was somehting to do with the upgrade
[16:12:49] <hashar>	 and there are bunch of pipeline related ones:  /var/lib/jenkins/plugins/pipe* 
[16:13:13] <wikibugs>	 (03CR) 10Thcipriani: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[16:13:30] * thcipriani looks in /var/lib/jenkins/plugins
[16:14:50] <thcipriani>	 hashar: working again after a reinstall: https://integration.wikimedia.org/ci/job/service-pipeline-test/2499/console
[16:15:03] <thcipriani>	 (see HttpMethod: POST
[16:15:05] <thcipriani>	 )
[16:15:36] <thcipriani>	 looking at the disabled plugins on contint1001, I don't see anything I recognize as in-use for the pipeline
[16:16:37] <hashar>	 OHHH
[16:16:38] <hashar>	 yeah
[16:16:39] <hashar>	 bah
[16:16:41] <hashar>	 my fault ;-\
[16:17:12] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Wikidata, and 4 others: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818 (10hashar) The job happens after a change has been merged (Zuul pipeline:...
[16:17:28] <hashar>	 thcipriani: so indeedI hae deleted that plugin since it was no more referred in any of the  jobs/*/config.xml files :\
[16:17:41] <hashar>	 and I obviously completely missed the pipeline/grovvy jobs bah
[16:17:42] <thcipriani>	 ah, well that explains it :)
[16:18:03] <hashar>	 I should have logged it I guess
[16:18:06] <hashar>	 mystery solved
[16:18:18] <hashar>	 James_F: yeah I noticed the ci-buster container :-]
[16:18:40] <hashar>	 SMalyshev: seems that the job passes now! https://integration.wikimedia.org/ci/job/wikidata-query-gui-build/239/ :]
[16:19:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Wikidata, and 4 others: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818 (10hashar) 05Open→03Resolved The job has produced https://gerrit.wikim...
[16:20:33] <hashar>	 James_F: and kudos for acting on the docker-registry issue yesterday
[16:20:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10greg)
[16:20:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201907): deployment-tin has disk space issues - https://phabricator.wikimedia.org/T166492 (10greg)
[16:21:04] <hashar>	 I posted my message on gerrit way past midnight iirc and I was not really in the best to offer any better advice :]
[16:21:16] <hashar>	 good news, apparently nothing broke following the containers rebuild
[16:21:45] <wikibugs>	 (03CR) 10Thcipriani: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[16:23:04] <James_F>	 hashar: Well, we're not using any of the new containers yet, so… :-)
[16:23:17] <hashar>	 paladox: thcipriani: I also commented about exposing Gerrit internal metrics up to prometheus/grafana at https://phabricator.wikimedia.org/T184086#5340821
[16:23:32] <hashar>	 James_F: oh you haven't updated them in jjb have you?
[16:23:35] <paladox>	 yeh, the prometheus plugin exposes metrics from gerrit core
[16:23:38] <hashar>	 James_F: which would explain the lack of breakage
[16:23:41] <paladox>	 e.g performance metrics && others
[16:23:44] <James_F>	 hashar: Nope, didn't push or merge that.
[16:24:09] <James_F>	 hashar: Because three hours in SRE had fixed the back-end issue from a backup they didn't think they had.
[16:24:25] <hashar>	 James_F: at some point I guess we might want to have the jjb job to always point to :latest and have the jobs to pull.  So that as soon a new image is deployed, it is live
[16:24:35] <James_F>	 The new images are *probably* mostly fine, but we should only upgrade to them one by one.
[16:25:03] <James_F>	 Should we? That means if we update an image and it's wrong we can't fix it quickly, we have to publish a new (revert) image.
[16:25:14] <hashar>	 James_F: as the head of infra of an important company told one day:  IT is not complicated, you need to keep it/IT simple and have a backup.
[16:25:25] <James_F>	 Ha.
[16:26:26] <hashar>	 paladox: cause I have seen metrics-reporter-jmx which is to expose Gerrit internal metrics to jmx.   Then I haven't noticed anything indicating that it is bubbled up by javamelody and ultimately to the javamelody prometheus endpoint
[16:27:01] <hashar>	 paladox: and I don't get what the other plugin ( metrics-reporter-prometheus ) does, it seems to add another endpoint we would need to configure with a bearer token and some config on prometheus side
[16:27:23] <paladox>	 metrics-reporter-prometheus exposes gerrit's internal metrics to prometheus
[16:27:28] <paladox>	 i've confirmed that :)
[16:27:33] <hashar>	 James_F: potentially with :latest, if something fail, we could update the jenkins job to point to the lasst known good tag
[16:27:37] <paladox>	 (i've installed the plugin and got a ton of metrics)
[16:27:48] <hashar>	 James_F: but yeah it is unclear to me what would be a better workflow :\
[16:28:20] <hashar>	 paladox: but are those gerrit internal metrics exposed in the javamelody prometheus endpoint (  https://gerrit.wikimedia.org/r/monitoring?format=prometheus  ? )
[16:28:30] <paladox>	 Nope
[16:28:36] <paladox>	 it's a new url you would fetch from
[16:28:37] <James_F>	 hashar: Yeah. :-(
[16:28:37] <hashar>	 so differnet url right?
[16:28:40] <paladox>	 yeh
[16:28:46] <hashar>	 James_F: that owuld be for later so :-]]]
[16:29:02] <paladox>	 @URL@plugins/@PLUGIN@/metrics URL.
[16:30:16] <hashar>	 :]
[16:31:23] <hashar>	 wrote on the task. Thank you paladox!
[16:31:25] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) I have quickly talked with @Paladox about it. He has tried the `metrics-reporter-prometh...
[16:31:34] <paladox>	 your welcome :)
[16:33:13] <wikibugs>	 (03PS1) 10Thcipriani: make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966
[16:34:08] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] make-wmf-branch: Add comments (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[16:35:38] <hashar>	 I am off for dinner/sleep etc. *wave*
[16:37:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Wikidata, and 4 others: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818 (10Smalyshev) The build seems to be fine now, thanks!
[16:43:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Wikidata, and 4 others: WDQS GUI deploy build fails - https://phabricator.wikimedia.org/T227818 (10hashar) Hurrah!  Thank you to have remaindered us about this task :-]
[16:44:20] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Trigger service-pipeline docker image builds of eventgate-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/523804 (https://phabricator.wikimedia.org/T226668) (owner: 10Ottomata)
[16:44:47] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Add phan dependencies for PageForms [integration/config] - 10https://gerrit.wikimedia.org/r/523894 (https://phabricator.wikimedia.org/T228155) (owner: 10Daimona Eaytoy)
[16:46:01] <wikibugs>	 (03Merged) 10jenkins-bot: Trigger service-pipeline docker image builds of eventgate-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/523804 (https://phabricator.wikimedia.org/T226668) (owner: 10Ottomata)
[16:46:07] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907): Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10Legoktm) Yay stretch! Ple...
[16:46:31] <wikibugs>	 (03Merged) 10jenkins-bot: Add phan dependencies for PageForms [integration/config] - 10https://gerrit.wikimedia.org/r/523894 (https://phabricator.wikimedia.org/T228155) (owner: 10Daimona Eaytoy)
[16:48:36] <James_F>	 !log Zuul: Register initial tests for eventgate-wikimedia T226668
[16:48:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:48:40] <stashbot>	 T226668: Factor out eventgate-wikimedia factory into its own gerrit repo and use it for deployment pipeline - https://phabricator.wikimedia.org/T226668
[16:49:09] <James_F>	 !log Zuul: [PageForms] Add phan dependencies on AdminLinks, Cargo, PageSchemas T228155
[16:49:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:49:12] <stashbot>	 T228155: Add phan to PageForms - https://phabricator.wikimedia.org/T228155
[16:50:16] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10Legoktm) >>! In T226240#5341629, @greg wrote: > I think we probably still want a mirror in product...
[16:50:37] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Parsoid, 10RESTBase, 10Core Platform Team Backlog (Later), 10Parsoid-Tests: Move Parsoid and RESTBase testing from Travis CI to our Jenkins - https://phabricator.wikimedia.org/T78410 (10Pchelolo) 05Open→03Declined For now I don't think it worths doing so. Th...
[16:57:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Pchelolo)
[16:57:53] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Operations, and 4 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10Pchelolo)
[16:58:09] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10dcausse)
[17:10:51] <wikibugs>	 10Phabricator: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10dbarratt)
[17:11:34] <wikibugs>	 10Phabricator, 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10dbarratt)
[17:18:20] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Reading-Infrastructure-Team-Backlog: Implement CI rules for new kartotherian repo - https://phabricator.wikimedia.org/T228170 (10Jdforrester-WMF) >>! In T228170#5341437, @Mholloway wrote: >>>! In T228170#5338071,...
[17:22:30] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10Aklapper) I don't see a need for this. Do you have an actual specific example / use case?  IMHO, most tasks are an idea and [if a report is waiting for further input (e.g. from its reporter or a third party) and can currently n...
[17:29:29] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10dbarratt) >>! In T228313#5342339, @Aklapper wrote: > I don't see a need for this. Do you have an actual specific example / use case?  Sure. T227733 T227595 T227544 T208175 etc.
[17:45:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Release Pipeline: Experiment with hosted kubernetes solutions for Beta - https://phabricator.wikimedia.org/T222820 (10greg) Moving to our current kanban board so @jeena can chat with @dduvall when he's back next week.
[17:56:44] <wikibugs>	 (03CR) 10Ottomata: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/523804 (https://phabricator.wikimedia.org/T226668) (owner: 10Ottomata)
[17:57:25] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma)
[17:58:01] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Add tests for scap.cli [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[17:58:03] <wikibugs>	 (03CR) 10Jforrester: "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/523804 (https://phabricator.wikimedia.org/T226668) (owner: 10Ottomata)
[17:58:06] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[18:00:26] <wikibugs>	 (03Merged) 10jenkins-bot: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma)
[18:01:17] <wikibugs>	 (03CR) 10jenkins-bot: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma)
[18:02:13] <wikibugs>	 (03Merged) 10jenkins-bot: Add tests for scap.cli [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[18:02:51] <wikibugs>	 (03CR) 10jenkins-bot: Add tests for scap.cli [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[18:04:02] <wikibugs>	 (03Merged) 10jenkins-bot: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[18:04:40] <wikibugs>	 (03CR) 10jenkins-bot: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli)
[18:20:05] <mutante>	 i used to be able to just "scap pull" on any mw host but now i have one where it fails with "mwscript  not found" ..eh...
[18:20:24] <mutante>	 also unhandled error  
[18:43:07] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[18:43:37] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[18:45:18] <thcipriani>	 hrm, verification of docker image failed..
[18:45:34] <thcipriani>	 mutante: oh? which host?
[18:46:00] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10Aklapper) Which underlying problem gets solved by being able to search for such "ideas" across all code repositories via such a new tag?
[18:53:20] <mutante>	 thcipriani: mw2181 - it was just reported for temperature issues originally   https://phabricator.wikimedia.org/T205240
[18:53:50] <thcipriani>	 I notice that mwscript is only included for appserver_canaries
[18:54:04] <thcipriani>	 since it's in scap::scripts
[18:54:09] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10dbarratt) >>! In T228313#5342701, @Aklapper wrote: > Which underlying problem gets solved by being able to search for such "ideas" across all code repositories via such a new tag? (If that is the expectation, don't know - it's...
[18:54:36] <mutante>	 thcipriani: something changed then. ack
[18:55:13] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10Krinkle) There is #Proposal which has been used from time to time to gather comments leading up to a potential Epic, Goal or TechCom-RFC.
[18:55:21] <mutante>	 i think we still need scap pull though
[18:57:05] <thcipriani>	 I'm trying to figure out if we suddently need mwscript on appservers, or if we suddenly don't have mwscript...i.e., scap v puppet
[18:57:42] <mutante>	 CalledProcessError: Command '/usr/local/bin/mwscript extensions/WikimediaMaintenance/refreshMessageBlobs.php' returned non-zero exit status 1
[18:57:45] <mutante>	 18:14:53 pull failed: <CalledProcessError> Command '/usr/local/bin/mwscript extensions/WikimediaMaintenance/refreshMessageBlobs.php' returned non-zero exit status 1
[18:57:55] <mutante>	 it's like maintenance stuff runs here.. hmm
[18:59:09] <Krinkle>	 mutante: depending on where you found that, that may've been be running 'scap pull' on a random app server.
[18:59:18] <Krinkle>	 which I realised doesn't work indeed due to that.
[18:59:30] <Krinkle>	 my intent was to clear opcache, which I then did manually.
[18:59:45] <mutante>	 yes, that's a random appservers. but this used to be the normal thing to do
[18:59:50] <thcipriani>	 refreshMessageBlobs is a new one
[18:59:55] <mutante>	 how else would we sync the code after hardware maintenance
[18:59:57] <Krinkle>	 afaik mwscript has never been on normal app servers, but scap-pull certainly did work by itself on random app servers
[18:59:59] <thcipriani>	 a new thing scap does
[19:00:03] <Krinkle>	 yeah, good point
[19:00:17] <Krinkle>	 seems simple enough to provision?
[19:01:05] <Krinkle>	 wonder what else varies between debug app server and normal app server.
[19:01:57] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht)
[19:02:01] <thcipriani>	 is there any harm is calling this from a normal appserver?
[19:02:07] <wikibugs>	 10Release-Engineering-Team, 10serviceops: 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10Dzahn)
[19:03:24] <Krinkle>	 thcipriani: refreshMessageBlobs?
[19:03:27] <thcipriani>	 yeah
[19:03:35] <Krinkle>	 thcipriani: well, yes and no.
[19:03:47] <Krinkle>	 thcipriani: It's not wiki or server specific. It calls to memcached to purge the RL msg cache.
[19:03:58] <Krinkle>	 doing that too often would start to cause perf issues.
[19:04:05] <Krinkle>	 But functionally fine and safe.
[19:04:07] <wikibugs>	 10Release-Engineering-Team, 10serviceops: 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10Jdforrester-WMF) Is this a circular dependency? `mwscript` (== `/usr/local/bin/mwscript`) is not part of the appserver base?
[19:04:35] <wikibugs>	 10Release-Engineering-Team, 10serviceops: 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10thcipriani) `refreshMessageBlobs` was added in T222539  One of two solutions: * Install `scap::scripts` on all appservers rather than canary appservers * rethink how this is incl...
[19:04:42] <Krinkle>	 Would be nice if we could make it only happen during scap sync-(file), and (from debug) for scap pull.
[19:06:20] <wikibugs>	 10Project-Admins: Create an 'idea' tag - https://phabricator.wikimedia.org/T228313 (10Aklapper) 05Open→03Declined Ah, thanks! There is a #proposal tag that I was not aware of or had forgotten again because too many random tags (thanks @Krinkle!). We have a `stalled` status. And you could use "Draft" or "Prop...
[19:06:51] <thcipriani>	 hrm. might be a bit messy to determine if it's being called from a debug server.
[19:07:35] <thcipriani>	 we obviously don't run it too often considering this update has been live for a couple weeks and no one has hit this issue :P
[19:09:31] <Krinkle>	 thcipriani: yeah, alternatively if we pushed to the debug server hte same way we push to normal servers, then scap pull wouldn't need it at all
[19:10:04] <Krinkle>	 e.g. scap sync debug1001 and scap sync-file debug1001 or sth like that.  That would also solve the issue of debug testing not being representative due to full pull vs selective push
[19:10:25] <wikibugs>	 (03CR) 1020after4: [C: 03+2] make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[19:11:00] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[19:11:18] <twentyafterfour>	 18:43:20 Unable to find image 'docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.6-s1' locally
[19:11:30] <apergos>	 on Monday we had asked for a backport and deploy of a fix to mediawiki that would fix a bug in master that broke wikidata entity dumps; this patch is now merged into master
[19:11:44] <James_F>	 twentyafterfour: Hmm.
[19:12:24] <apergos>	 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/523412/    
[19:12:44] <James_F>	 twentyafterfour: 0.2.6-s1 is what jjb should be using; I built composer-test-hhvm:0.2.6-s2 last night but didn't switch jobs over as it wasn't needed.
[19:13:11] <James_F>	 apergos: Did it not make it into wmf.14?
[19:13:22] <apergos>	 not merged until today
[19:13:27] <apergos>	 so no it did not make the branch cut
[19:13:36] <James_F>	 Fun.
[19:13:48] <James_F>	 Anyway, sounds fine, want me to do it?
[19:14:00] <apergos>	 hoo ^^
[19:14:18] <James_F>	 (Tsk, no Phab task?)
[19:14:43] <apergos>	 there was but aaron's new patch maybe doesn't have the bug on the commit message
[19:14:54] <Krinkle>	 This is why we need QA: https://twitter.com/erinfranmc/status/1148986961207730176?s=20
[19:15:20] <apergos>	 https://phabricator.wikimedia.org/T228104   this is the task
[19:15:53] <apergos>	 hoo, you ok with James_F doing the backport?
[19:16:20] <hoo>	 Sure thing
[19:16:24] <James_F>	 Cool.
[19:16:30] <apergos>	 thank you in advance
[19:16:38] <James_F>	 Any time.
[19:17:31] <thcipriani>	 Krinkle: other question, do we want this to happen when we run scap pull at all? Nothing has been sync'd anywhere except for the debug server and the deployment host: how are the blobs repopulated after the message is purged?
[19:17:50] <legoktm>	 Krinkle: sounds like an integration test was needed there :p
[19:18:30] <Krinkle>	 thcipriani: indeed, if we make it work differently for debug, it does not need to happen on pull.
[19:18:47] <Krinkle>	 it needs to happen at least once after localisation data has been deployed.
[19:19:04] <Krinkle>	 if the server was down, gets new code and is then pooled, it's fine.
[19:19:18] <Krinkle>	 it purges something in memc, not on-server
[19:19:39] <thcipriani>	 we have it triggered after scap sync
[19:20:10] <thcipriani>	 right, so if it doesn't purge anything on that particular server, we don't want it to run after a scap pull to mwdebug1002, say?
[19:22:46] <wikibugs>	 10MediaWiki-Releasing, 10Patch-For-Review: Review 1.33.1 tarball - https://phabricator.wikimedia.org/T227599 (10MarkAHershberger) From meeting: * Write documentation * Mukunda reviews code * See what the issue is with composer  * Mark ran composer because he read reports on Support Desk about missing composer...
[19:23:45] <apergos>	 I feel uneasy about backporting as far back as 13 with this change, because loadbalancer and etc...
[19:25:14] <wikibugs>	 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops, 10Patch-For-Review, 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10Jdforrester-WMF) We're seeing this happening now on contint...
[19:27:17] <hoo>	 James_F: greg-g gave us the go (in -operations)… we need to go to wmf13 and wmf14
[19:27:42] <James_F>	 I thought it was already back-ported to wmf.13?
[19:27:55] <apergos>	 you did 14, no?
[19:27:59] <James_F>	 Oh, a different fix was backported?
[19:28:06] <James_F>	 Am doing .14, MW tests take an age.
[19:28:14] <hoo>	 No, it's broken on wmf13
[19:29:08] <apergos>	 13 went around late enough last week that the part of the run that could break was already past
[19:29:21] <apergos>	 so it wasn't noticed until this week
[19:29:38] <James_F>	 Ah.
[19:30:00] <James_F>	 (This is the wrong channel for production change testing, BTW.)
[19:31:04] <wikibugs>	 10MediaWiki-Releasing, 10Patch-For-Review: Review 1.33.1 tarball - https://phabricator.wikimedia.org/T227599 (10Legoktm) >>! In T227599#5342829, @MarkAHershberger wrote: > * See what the issue is with composer >  * Mark ran composer because he read reports on Support Desk about missing composer libs.  Were the...
[19:31:07] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10MediaWiki-Release-Tools, 10MediaWiki-Releasing (Workflow Improvements): Bot to upgrade the different bits of mw.o for tarball releases - https://phabricator.wikimedia.org/T217964 (10CCicalese_WMF)
[19:35:08] <wikibugs>	 10Release-Engineering-Team, 10serviceops: 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10thcipriani) >>! In T228328#5342778, @thcipriani wrote: > `refreshMessageBlobs` was added in T222539 >  > One of two solutions: > * Install `scap::scripts` on all appservers rathe...
[19:39:24] <wikibugs>	 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops, 10Patch-For-Review, 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10thcipriani) For that particular image I can recreate locall...
[19:41:45] <mutante>	 i suddenly see a exclamation mark in phab next to my notifcations that i never had before
[19:41:52] <mutante>	 and it says "unresolved setup issues"
[19:41:54] <mutante>	 twentyafterfour: 
[19:42:21] <mutante>	 i mean it says there is 1 of them
[19:44:52] <wikibugs>	 (03PS1) 10Thcipriani: refreshMessageBlobs: don't run on scap pull [tools/scap] - 10https://gerrit.wikimedia.org/r/523999 (https://phabricator.wikimedia.org/T228328)
[19:48:29] <wikibugs>	 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO, 10MediaWiki-Release-Tools, 10MediaWiki-Releasing (Workflow Improvements): Bot to upgrade the different bits of mw.o for tarball releases - https://phabricator.wikimedia.org/T217964 (10greg)
[20:18:18] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[20:19:45] <urandom>	 what does "ERROR: Could not determine Gerrit ChangeID from Environment. Aborting." mean?
[20:19:53] <urandom>	 https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/400/console
[20:20:29] <urandom>	 is that because I pushed a new tag without a corresponding changeset? 
[20:21:12] <urandom>	 i.e. is it because there have been no new gerrit changesets?
[20:35:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Workboards (Clinic Duty Team), and 4 others: install.php --with-extensions silently ignores extensio... - https://phabricator.wikimedia.org/T225512
[20:39:21] <andrewbogott>	 thcipriani: Last week we discussed moving those deployment-prep VMs on Friday rather than Monday — it looks like things are set up so I should be able to do that.  Do you want to coordinate as I move each VM, or should I just send an email to -releng warning about an outage, or ???
[20:46:49] <thcipriani>	 andrewbogott: an email about the outage is probably easiest to coordinate. If you can send that email to -releng, and we can broadcast to other lists/modify channel topic
[20:47:02] <andrewbogott>	 ok!
[20:51:41] <wikibugs>	 (03CR) 1020after4: [C: 03+2] "recheck" [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[20:52:01] <andrewbogott>	 thcipriani: sent
[20:52:23] <wikibugs>	 (03Merged) 10jenkins-bot: make-wmf-branch: Add comments [tools/release] - 10https://gerrit.wikimedia.org/r/523966 (owner: 10Thcipriani)
[21:06:27] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Jdforrester-WMF)
[21:06:37] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Jdforrester-WMF)
[21:07:16] <hauskatze>	 Unable to find image 'docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.6-s1' locally
[21:07:25] <hauskatze>	 https://integration.wikimedia.org/ci/job/composer-hhvm-docker/454/console
[21:07:28] <hauskatze>	 :|
[21:10:32] <wikibugs>	 (03CR) 1020after4: [C: 03+1] refreshMessageBlobs: don't run on scap pull [tools/scap] - 10https://gerrit.wikimedia.org/r/523999 (https://phabricator.wikimedia.org/T228328) (owner: 10Thcipriani)
[21:11:25] <James_F>	 hauskatze: Recheck it and it'll probably work.
[21:11:36] <James_F>	 But yeah, meh.
[21:13:16] <hauskatze>	 James_F: I'll retry. Third time's the charm right? :)
[21:14:23] <James_F>	 One can but hope.
[21:14:37] <hauskatze>	 It passed
[21:14:45] <hauskatze>	 zuul was kind with me this time
[21:36:19] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins: Increase TTL of failed builds - https://phabricator.wikimedia.org/T228158 (10hashar) The XML configuration for a job looks like: ` lang=xml <project>   <properties>     <jenkins.model.BuildDiscarderProperty>       <strategy class="hudson.tasks.LogRotator">...
[21:42:01] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10greg)
[21:47:25] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10Jdforrester-WMF)
[21:59:20] <hauskatze>	 James_F: so wikimedia/iegreview repo fails intermitently on zuul
[21:59:32] <hauskatze>	 I've issued 'recheck' several times to a patch now
[22:00:09] <James_F>	 Meh.
[22:04:18] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10Catrope)
[22:04:26] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T220739 (10Catrope)
[22:18:34] <apergos>	 James_F: signing off, things still look ok
[22:33:24] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[22:41:17] <wikibugs>	 (03PS1) 10Catrope: Add TheWikipediaLibrary extension [tools/release] - 10https://gerrit.wikimedia.org/r/524052 (https://phabricator.wikimedia.org/T132084)
[22:42:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add TheWikipediaLibrary extension [tools/release] - 10https://gerrit.wikimedia.org/r/524052 (https://phabricator.wikimedia.org/T132084) (owner: 10Catrope)
[22:44:41] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Security: Set "auth.lock-config" to true - https://phabricator.wikimedia.org/T220670 (10Aklapper)
[22:44:45] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Security: Set "auth.lock-config" to true - https://phabricator.wikimedia.org/T220670 (10Aklapper)
[22:44:54] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907): Set "auth.lock-config" to true - https://phabricator.wikimedia.org/T220670 (10Aklapper) Making this task public.
[22:45:02] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[23:05:46] <wikibugs>	 (03CR) 10Catrope: "recheck" [tools/release] - 10https://gerrit.wikimedia.org/r/524052 (https://phabricator.wikimedia.org/T132084) (owner: 10Catrope)
[23:08:26] <ejegg|afk>	 lots of filing docker builds in fundraising repos too :(
[23:09:55] <wikibugs>	 10Project-Admins, 10Research: create tags for Research - https://phabricator.wikimedia.org/T228350 (10leila)
[23:57:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - requesting bureaucrat rights - https://phabricator.wikimedia.org/T228352 (10DannyS712)