[00:14:56] <wikibugs>	 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10greg) p:05Triage→03High (setting to high to reflect reality)
[00:16:11] <wikibugs>	 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10greg) @Krenair / @hashar / @jcrespo : What are the next steps here?
[00:16:15] <greg-g>	 jdlrobson: ^ that's the task to watch
[01:39:28] <wikibugs>	 (03PS1) 10Thcipriani: Add support for Gerrit Comments [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/492225
[01:44:18] <wikibugs>	 (03PS2) 10Thcipriani: Add support for Gerrit Comments [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/492225
[01:47:15] <wikibugs>	 (03PS3) 10Thcipriani: Add support for Gerrit Comments [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/492225
[01:48:31] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10mmodell)  >>! In T213595#4972704, @greg wrote: > Did that ^ happen?  I'm not aware of any security patches that landed in 1.32.1
[02:05:22] <wikibugs>	 (03PS4) 10Thcipriani: Improve Deployment Pipeline/Gerrit feedback [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/492225
[03:46:53] <paladox>	 thcipriani oh there seems to be 2.15.11 (being released soon)
[03:46:54] <paladox>	 https://gerrit-review.googlesource.com/c/gerrit/+/215232
[04:46:35] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Reproducible exception: Query "ManiphestTaskQuery" failed to return a value from getPagingValueMap() for column "rank". - https://phabricator.wikimedia.org/T198661 (10Aklapper) >>! In T198661#4854654, @Aklapper wrote: > Should test again in our instance //once we've pulled...
[04:49:19] <wikibugs>	 10Phabricator (2019-02-20), 10Upstream: Custom fields for task subtypes fatal on pages > 1 with: Query "ManiphestTaskQuery" failed to return a value from getPagingValueMap() - https://phabricator.wikimedia.org/T168556 (10Aklapper) 05Stalled→03Resolved I cannot reproduce the problem anymore after the #phab-...
[04:49:54] <wikibugs>	 10Phabricator (Upstream), 10Upstream: #1054: Unknown column 'ft_doc.epochModified' in 'order clause' - https://phabricator.wikimedia.org/T187604 (10Aklapper) For the records, the problem still happens after the #phab-2019-02-20 update, as https://secure.phabricator.com/T13091
[04:52:16] <wikibugs>	 10Phabricator (2019-02-20), 10Security, 10Upstream: Creating a sub-task of a security issue (via "Edit related tasks" menu) does not automatically protect the task as Security - https://phabricator.wikimedia.org/T175072 (10Aklapper) 05Open→03Resolved >>! In T175072#4815209, @Aklapper wrote: > Worth to ch...
[04:53:37] <wikibugs>	 10Phabricator (2019-02-20), 10Release-Engineering-Team (Kanban), 10Upstream, 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10Aklapper)
[05:50:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Joe...
[06:16:30] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Beta-Feature: [BetaFeatures] sql/create_counts.sql : Error: 1071 Specified key was too long; max key length is 767 bytes - https://phabricator.wikimedia.org/T193533 (10Liuxinyu970226)
[06:16:48] <wikibugs>	 10Release-Engineering-Team (Kanban), 10BetaFeatures: [BetaFeatures] sql/create_counts.sql : Error: 1071 Specified key was too long; max key length is 767 bytes - https://phabricator.wikimedia.org/T193533 (10Liuxinyu970226)
[06:17:06] <wikibugs>	 (03PS1) 10Zoranzoki21: Archive RelatedSites extension [integration/config] - 10https://gerrit.wikimedia.org/r/492240 (https://phabricator.wikimedia.org/T214458)
[06:21:02] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-CodeReview: CodeReview extension archive scope definition - https://phabricator.wikimedia.org/T216219 (10Liuxinyu970226) Is there really having problems to convert these logs to e.g. Gerrit commits?
[06:26:33] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Error: 1071 Specified key was too long; max key length is 767 bytes - https://phabricator.wikimedia.org/T193222 (10Liuxinyu970226)
[06:32:02] <wikibugs>	 10Project-Admins: Rename #Wikimedia-production-error to something less generic? - https://phabricator.wikimedia.org/T216795 (10Aklapper) p:05Triage→03Lowest
[06:33:03] <wikibugs>	 10Project-Admins: Rename #Wikimedia-production-error to something less generic? - https://phabricator.wikimedia.org/T216795 (10Aklapper)
[06:39:56] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations, 10Patch-For-Review: Upgrade scap debian package to 3.9.0-1 - https://phabricator.wikimedia.org/T216666 (10greg) 05Open→03Resolved a:03fsero Thanks!
[07:39:56] <wikibugs>	 10Release-Engineering-Team (Kanban): docker-registry.wikimedia.org returns 404 - https://phabricator.wikimedia.org/T216799 (10ItSpiderman)
[07:43:05] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Archive GooglePlusOne extension [integration/config] - 10https://gerrit.wikimedia.org/r/491982 (https://phabricator.wikimedia.org/T216675) (owner: 10Zoranzoki21)
[07:44:22] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Archive RelatedSites extension [integration/config] - 10https://gerrit.wikimedia.org/r/492240 (https://phabricator.wikimedia.org/T214458) (owner: 10Zoranzoki21)
[07:44:47] <wikibugs>	 (03Merged) 10jenkins-bot: Archive GooglePlusOne extension [integration/config] - 10https://gerrit.wikimedia.org/r/491982 (https://phabricator.wikimedia.org/T216675) (owner: 10Zoranzoki21)
[07:46:09] <wikibugs>	 (03Merged) 10jenkins-bot: Archive RelatedSites extension [integration/config] - 10https://gerrit.wikimedia.org/r/492240 (https://phabricator.wikimedia.org/T214458) (owner: 10Zoranzoki21)
[07:46:31] <wikibugs>	 (03PS1) 10Hashar: Add debian-glue job to operations/debs/envoyproxy [integration/config] - 10https://gerrit.wikimedia.org/r/492254 (https://phabricator.wikimedia.org/T215810)
[07:46:43] <wikibugs>	 (03PS2) 10Hashar: Add debian-glue job to operations/debs/envoyproxy [integration/config] - 10https://gerrit.wikimedia.org/r/492254 (https://phabricator.wikimedia.org/T215810)
[07:47:20] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "That is just for your information :)" [integration/config] - 10https://gerrit.wikimedia.org/r/492254 (https://phabricator.wikimedia.org/T215810) (owner: 10Hashar)
[07:49:13] <wikibugs>	 (03Merged) 10jenkins-bot: Add debian-glue job to operations/debs/envoyproxy [integration/config] - 10https://gerrit.wikimedia.org/r/492254 (https://phabricator.wikimedia.org/T215810) (owner: 10Hashar)
[07:58:37] <wikibugs>	 (03PS1) 10Hashar: Register mediawiki/extensions/NewsTicker [integration/config] - 10https://gerrit.wikimedia.org/r/492255
[07:58:49] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Register mediawiki/extensions/NewsTicker [integration/config] - 10https://gerrit.wikimedia.org/r/492255 (owner: 10Hashar)
[08:00:14] <wikibugs>	 (03Merged) 10jenkins-bot: Register mediawiki/extensions/NewsTicker [integration/config] - 10https://gerrit.wikimedia.org/r/492255 (owner: 10Hashar)
[08:00:27] <wikibugs>	 (03PS1) 10Hashar: Add debian-glue to operations/debs/tideways-xhprof [integration/config] - 10https://gerrit.wikimedia.org/r/492256 (https://phabricator.wikimedia.org/T176916)
[08:01:14] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add debian-glue to operations/debs/tideways-xhprof [integration/config] - 10https://gerrit.wikimedia.org/r/492256 (https://phabricator.wikimedia.org/T176916) (owner: 10Hashar)
[08:04:09] <wikibugs>	 (03Merged) 10jenkins-bot: Add debian-glue to operations/debs/tideways-xhprof [integration/config] - 10https://gerrit.wikimedia.org/r/492256 (https://phabricator.wikimedia.org/T176916) (owner: 10Hashar)
[08:12:33] <wikibugs>	 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Kanban): Setup CI for operations/software/gerrit - https://phabricator.wikimedia.org/T189549 (10hashar)
[08:12:36] <wikibugs>	 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review: Add CI to all operations/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180330 (10hashar)
[08:16:36] <wikibugs>	 10Continuous-Integration-Config, 10Security-Team: Add tests/CI to wikimedia/security/tooling - https://phabricator.wikimedia.org/T216801 (10hashar)
[08:17:03] <wikibugs>	 10Continuous-Integration-Config, 10Tracking: Add CI to all Gerrit repositories - https://phabricator.wikimedia.org/T180317 (10hashar)
[08:17:05] <wikibugs>	 10Continuous-Integration-Config, 10Security-Team: Add tests/CI to wikimedia/security/tooling - https://phabricator.wikimedia.org/T216801 (10hashar)
[08:23:22] <wikibugs>	 (03CR) 10Nikerabbit: [C: 03+1] Make the Translate extensions coverage report public [integration/config] - 10https://gerrit.wikimedia.org/r/491446 (owner: 10Thiemo Kreuz (WMDE))
[08:33:01] <wikibugs>	 (03PS1) 10Hashar: Run tox for wikimedia/security/tooling [integration/config] - 10https://gerrit.wikimedia.org/r/492261 (https://phabricator.wikimedia.org/T216801)
[08:33:49] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Run tox for wikimedia/security/tooling [integration/config] - 10https://gerrit.wikimedia.org/r/492261 (https://phabricator.wikimedia.org/T216801) (owner: 10Hashar)
[08:36:30] <wikibugs>	 (03Merged) 10jenkins-bot: Run tox for wikimedia/security/tooling [integration/config] - 10https://gerrit.wikimedia.org/r/492261 (https://phabricator.wikimedia.org/T216801) (owner: 10Hashar)
[08:37:10] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Security-Team, 10Patch-For-Review: Add tests/CI to wikimedia/security/tooling - https://phabricator.wikimedia.org/T216801 (10hashar) a:03hashar
[09:12:27] <wikibugs>	 (03PS2) 10Hashar: Make the Translate extensions coverage report public [integration/config] - 10https://gerrit.wikimedia.org/r/491446 (owner: 10Thiemo Kreuz (WMDE))
[09:12:29] <wikibugs>	 (03PS1) 10Hashar: gitignore .idea/ [integration/config] - 10https://gerrit.wikimedia.org/r/492268
[09:12:40] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] gitignore .idea/ [integration/config] - 10https://gerrit.wikimedia.org/r/492268 (owner: 10Hashar)
[09:13:55] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "I am not familiar with that job, but lets give it a try :)" [integration/config] - 10https://gerrit.wikimedia.org/r/491446 (owner: 10Thiemo Kreuz (WMDE))
[09:14:10] <wikibugs>	 (03Merged) 10jenkins-bot: gitignore .idea/ [integration/config] - 10https://gerrit.wikimedia.org/r/492268 (owner: 10Hashar)
[09:15:28] <wikibugs>	 (03Merged) 10jenkins-bot: Make the Translate extensions coverage report public [integration/config] - 10https://gerrit.wikimedia.org/r/491446 (owner: 10Thiemo Kreuz (WMDE))
[09:28:21] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "On contint1001:" [integration/config] - 10https://gerrit.wikimedia.org/r/491446 (owner: 10Thiemo Kreuz (WMDE))
[09:41:06] <wikibugs>	 10Release-Engineering-Team (Kanban): docker-registry.wikimedia.org returns 404 - https://phabricator.wikimedia.org/T216799 (10hashar) a:05hashar→03None https://docker-registry.wikimedia.org/v2/ works for me  https://docker-registry.wikimedia.org/v2/_catalog is super slow though.
[09:45:03] <wikibugs>	 10Release-Engineering-Team (Kanban): docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10hashar)
[10:29:20] <wikibugs>	 10Release-Engineering-Team (Kanban), 10TimedMediaHandler, 10User-zeljkofilipin: The first Selenium test for TimedMediaHandler - https://phabricator.wikimedia.org/T214480 (10zeljkofilipin) I've assigned it to me for now, I'll create a sample patch today or on Monday.
[11:01:36] <wikibugs>	 10Project-Admins: Create a "Moegirlpedia" tag in Phabricator? - https://phabricator.wikimedia.org/T215688 (10Aklapper)
[11:01:40] <wikibugs>	 10Project-Admins: Create a "Moegirlpedia" tag in Phabricator? - https://phabricator.wikimedia.org/T215688 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #Moegirlpedia has been created: https://phabricator.wikimedia.org/project/view/3901/  Please encourage interested people to visit th...
[11:05:52] <wikibugs>	 10Project-Admins: Create project for GraphQL extension - https://phabricator.wikimedia.org/T216310 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #MediaWiki-extensions-GraphQL has been created: https://phabricator.wikimedia.org/project/view/3902/ (I've also edited the description of #...
[11:06:49] <wikibugs>	 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10Aklapper) @ashley: Could you reply to my last question please?
[11:10:45] <wikibugs>	 10Project-Admins: Create phabricator project for Extension:Blackout - https://phabricator.wikimedia.org/T203227 (10Aklapper) p:05Triage→03Lowest I don't see active development in https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/Blackout hence I propose declining this for the time being until T...
[11:12:40] <wikibugs>	 10Project-Admins: Create component project for Status tool - https://phabricator.wikimedia.org/T216082 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #status has been created: https://phabricator.wikimedia.org/project/view/3903/  Please encourage interested people to visit the project...
[11:39:39] <Krinkle>	 hashar: o/
[12:11:57] <wikibugs>	 10Project-Admins: Create component project for Status tool - https://phabricator.wikimedia.org/T216082 (10MGChecker) Wouldn't be #Tool-status the better choice? #status seems like a quite generic tag name to spend here.
[12:15:54] <hashar>	 Krinkle: hello sorry
[12:16:03] <hashar>	 I had lunch break
[12:34:26] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[12:36:13] <hashar>	 bah
[12:37:30] <icinga-wm>	 ACKNOWLEDGEMENT - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] amusso spam of changes. No worries. https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[12:38:59] <hashar>	 so my loong duty 
[12:39:06] <hashar>	 is webdriver.io exploding nodejs somehow :/
[12:39:17] <hashar>	 and I found out on test failure we take two screenshots
[12:39:48] <hashar>	 one due to wdio options.screenshotPath  which when set cause wdio to automatically take a screenshot ( eg:  ERROR-Chrome-<some date>.png )
[12:40:18] <hashar>	 the other is our afterTest hook which also browser.saveScreenshot() ;D
[12:40:43] <hashar>	 and
[12:41:02] <hashar>	 I got ffmpeg processes left behind from child_process.spawn() grr
[12:57:34] <wikibugs>	 10Release-Engineering-Team, 10Analytics, 10EventBus, 10MediaWiki-Core-Testing, and 4 others: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins - https://phabricator.wikimedia.org/T216069 (10hashar) 05Open→03Resolved a:03hashar Root cause figured out. I will fill another task to cover the weird w...
[13:19:03] <wikibugs>	 10Release-Engineering-Team (Kanban): Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10ItSpiderman)
[13:21:36] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[13:22:04] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble: Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10Reedy)
[13:28:37] <wikibugs>	 10Project-Admins: Create component project for Status tool - https://phabricator.wikimedia.org/T216082 (10Aklapper) We could always rename if too many wrong tickets get created, but its description and that it's a subproject should help? (In theory, if people see that... :-/ )
[13:36:32] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[13:38:55] <Reedy>	 !log restarting zuul because nothing is being run
[13:38:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:45:15] <hashar>	 Reedy: restart ?
[13:45:16] <hashar>	 no
[13:45:28] <hashar>	 Reedy: dont restart it please
[13:45:39] <Reedy>	 Well, your docs are wrong then
[13:45:43] <Reedy>	 Graceful
[13:45:44] <Reedy>	 A plain "restart" is graceful.
[13:45:44] <Reedy>	 ssh contint1001.wikimedia.org
[13:45:44] <Reedy>	 /usr/sbin/service zuul restart && tail -f -n100 /var/log/zuul/zuul.log
[13:45:47] <Reedy>	 https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
[13:45:51] <wikibugs>	 10Project-Admins: Create a "Moegirlpedia" tag in Phabricator? - https://phabricator.wikimedia.org/T215688 (10Zoglun) @Aklapper Cool! Thank you!
[13:46:03] <hashar>	 Reedy: yeah which mean it does not "fix" anything anyway
[13:46:12] <hashar>	 the graceful restart wait for jobs to complete
[13:46:17] <hashar>	 save the current state
[13:46:19] <hashar>	 shutdown
[13:46:22] <hashar>	 start again
[13:46:25] <hashar>	 load the saved state
[13:46:29] <hashar>	 and rerun all the jobs
[13:46:47] <hashar>	 which means it has to process again anything that was stalling it
[13:47:10] <hashar>	 which is to craft the merge commits for the long chain of operations/puppet.git patches
[13:47:29] <hashar>	 so each time your restart it, that start the process from zero
[13:48:19] <hashar>	 you can look at /var/log/zuul/merger-debug.log  on contint1001 and contint2001
[13:49:06] <hashar>	 that is similar to https://phabricator.wikimedia.org/T151089
[14:12:50] <wikibugs>	 (03CR) 10Lars Wirzenius: [C: 03+1] "I don't know enough about the intended purpose to judge this properly, and I don't know Groovy, but I didn't see anything obviously bad. F" (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/492225 (owner: 10Thcipriani)
[14:23:29] <hashar>	 ah madness continue
[14:23:37] <hashar>	 nodejs leaving processes behind :((
[14:31:08] <Zppix>	 hashar:  can i have a patch deployed i would schedule for swat but the times for swat are hard for me find time in my schedule, my patch should be painless to deploy and should take just a few seconds
[14:32:01] <hashar>	 Zppix: there is no deployment on friday unless critical! ;D
[14:32:18] <hashar>	 for small and simple patches, I guess they can be added to SWAT and we should be able to verify them by ourselves
[14:32:40] <Zppix>	 hashar:  do i need to leave a note saying i cannot be there ?
[14:36:08] <hashar>	 Zppix: yeah I guess a note will be sufficient ;)
[14:36:33] <Zppix>	 hashar:  okay will do thanks antoine
[14:50:59] <_joe_>	 hashar: jenkins is super slow, any idea why?
[14:51:38] <hashar>	 _joe_: what is slow? ;)
[14:51:55] <hashar>	 note that an hour or so ago there was a long chain of operations/puppet patches
[14:52:03] <hashar>	 and zuul does not quite nicely handle that case
[14:52:16] <hashar>	 (it tries to generate a lot of merge commit which takes a while)
[14:52:17] <_joe_>	 no this is just a patch on a mw extension
[14:52:38] <_joe_>	 usually I get a feedback in 3/4 minutes, this is there since 9
[14:52:50] <_joe_>	 also it should fail pretty soon in the chain
[14:52:56] <_joe_>	 there is a formatting error
[14:53:09] <hashar>	 ah yeah there are some stalled changes
[15:27:40] <wikibugs>	 10Project-Admins, 10Collection, 10OCG-General: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312 (10ovasileva)
[15:41:58] <wikibugs>	 (03PS1) 10Sophivorus: Add Extension:WikipediaExtracts [integration/config] - 10https://gerrit.wikimedia.org/r/492342 (https://phabricator.wikimedia.org/T149765)
[15:55:34] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:16:14] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble: Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10hashar)
[16:34:46] <wikibugs>	 10Release-Engineering-Team (Backlog), 10Quibble: Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10greg) a:05hashar→03None
[16:38:42] <wikibugs>	 10Release-Engineering-Team (Backlog), 10Quibble: Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10hashar) Indeed #Quibble is hardcoded for Gerrit. Precisely, Quibble is a test runner intended for Wikimedia CI and for its Gerrit installation. It is also possible to...
[16:39:06] <wikibugs>	 10Release-Engineering-Team (Backlog), 10Quibble: Using Quibble with different backend than gerrit - https://phabricator.wikimedia.org/T216819 (10hashar) Side question: are you familiar with python? I am definitely willing to review patches proposals :-]
[16:43:53] <wikibugs>	 10Release-Engineering-Team, 10docker-pkg: docker-pkg fails with "Is python3-apt installed?" - https://phabricator.wikimedia.org/T216836 (10Krinkle)
[16:48:53] <paladox>	 !log created "block-users" group for https://bugs.chromium.org/p/gerrit/issues/detail?id=10507 (reported by a wmf user)
[16:48:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:50:21] <wikibugs>	 (03PS1) 10Paladox: Block read on refs/* to users in block-users group [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/492348
[16:54:37] <paladox>	 thcipriani around to review https://gerrit.wikimedia.org/r/#/c/All-Projects/+/492348/ please? :)
[16:58:38] * paladox wants to try this out at somepoint https://gerrithub.io/q/project:GerritForge/plugins_multi-site have a multi master setup for gerrit :)
[17:19:02] <wikibugs>	 (03PS1) 10Mark Bergsma: WiP: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352
[17:20:32] <wikibugs>	 (03PS2) 10Mark Bergsma: WiP: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352
[17:22:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WiP: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma)
[17:32:38] <urandom>	 Blubber question related to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/kask/+/490683/4/.pipeline/blubber.yaml: Is the relative path in that entrypoint expected to work?  Or should be installing/copying the binary onto `$PATH`?
[17:33:41] <urandom>	 Hrmm.... wasn't there a way to run this through blubberoid to see what the resulting Dockerfile looked like?
[17:33:55] <liw>	 urandom, there is
[17:34:00] <liw>	 urandom, just a sec
[17:34:36] <liw>	 curl -T blubberoid.json -H "Content-Type: application/json" https://blubberoid.wikimedia.org/v1/test
[17:34:49] <liw>	 where blubberoid.json is the blubber.yaml file converted into JSON
[17:35:02] <urandom>	 yikes
[17:35:10] <urandom>	 OK
[17:35:25] <liw>	 and the final /test is the variant name you'd give blubber if you ran it directly
[17:36:21] <liw>	 thcipriani, I don't think use of blubberoid.wikimedia.org is documented under https://wikitech.wikimedia.org/wiki/Blubber - should it be?
[17:36:47] <thcipriani>	 (/me reading scrollback, just got out of meeting)
[17:37:15] <thcipriani>	 liw: the idea was that while CI would use blubberoid, the blubber binary on releases is mean for local development
[17:37:19] <thcipriani>	 *meant
[17:37:30] <thcipriani>	 https://releases.wikimedia.org/blubber/
[17:37:51] <liw>	 thcipriani, sure, but I'd assume people would like to try Blubber via the blubberoid service without having to install anything?
[17:40:47] <thcipriani>	 liw: there's a lot of discussion about this on the task here https://phabricator.wikimedia.org/T212251 and likely in the pipeline cabal meeting notes, in general there are folks against using the blubber service for local development for various reasons (although it's true that you can use it :))
[17:42:11] <liw>	 thcipriani, check
[17:42:26] <thcipriani>	 urandom: to answer your question, relative paths work. The "lives: {in: <path>}" is the path to which it should be relative
[17:42:44] <urandom>	 how does the source get there, btw?
[17:43:36] <urandom>	 is the source exported to that location when the build is started?
[17:43:50] <thcipriani>	 so in the case of your patch, the prep variant copies in the source, and runs "make build" then the "copies prep" line copies the resulting binary over to the final production container.
[17:44:23] <urandom>	 ??
[17:44:36] <thcipriani>	 right, so before "make build" the "requirements: [.]" in your blubber file will copy in all the source before starting make build
[17:45:23] <urandom>	 copy it to the "lives in"?
[17:45:40] <thcipriani>	 yep
[17:46:14] <urandom>	 and the "copies: prep" is just repeating all that for production?
[17:46:39] <urandom>	 the copy to "lives in" and the build?
[17:46:41] <thcipriani>	 the "copies: [prep]" is copying from the prep container, rather than your local filesystem
[17:46:53] <thcipriani>	 s/prep container/prep container image/
[17:47:24] <thcipriani>	 so any artifacts that are generated by the commands run in the "prep" variant will be copied to the production variant as well
[17:47:51] <thcipriani>	 in this case, since you are running "make build" and artifacts created by that command will be copied to the production image (in addition to the source)
[17:48:44] <urandom>	 I guess this is for the likes of node software where it's convention to just run it from the working directory
[17:49:02] <urandom>	 where the source and release artifacts are one in the same
[17:49:05] <thcipriani>	 yeah
[17:49:27] <thcipriani>	 we have a change happening soon so that you can specify *what* to copy from the prep container image Soon™
[17:49:48] <thcipriani>	 since we also use this for Blubber itself and we don't need the source code in the production image there either.
[17:50:11] <thcipriani>	 just the statically compiled binary should be copied from the prep image (which is probably ideal for kask as well_)
[17:50:11] <urandom>	 ok
[17:50:17] <urandom>	 yeah
[17:50:31] <urandom>	 so...what is the point of prep?
[17:50:53] <thcipriani>	 the idea with prep is that you can use a different base image, or install, say, build-essential
[17:51:19] <thcipriani>	 and use that to build binaries, that you can then just copy to the final container (which won't have build-essential installed)
[17:51:31] <urandom>	 ok
[17:52:09] <thcipriani>	 it's something docker calls a "multi-stage build" or "build pattern"
[17:52:17] <thcipriani>	 https://docs.docker.com/develop/develop-images/multistage-build/
[17:53:17] <urandom>	 so....going back to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/kask/+/490683/4/.pipeline/blubber.yaml, is this supposed to be excluding the Go packages installed via apt?
[17:53:25] <urandom>	 in the production image?
[17:54:05] <urandom>	 or perhaps my question should more specifically be about the relationship/difference between build and prep
[17:54:17] <urandom>	 if I could articulate that correctly
[17:55:19] <thcipriani>	 that blubber file would result in a final image that would not include the "go" binary but would have the compiled "kask" binary
[17:56:12] <thcipriani>	 the "production" variant would be based on wikimedia-stretch, but the "prep" variant would be based on "golang:1.10-stretch"
[17:57:02] <urandom>	 OK, I think I have it
[17:57:17] <thcipriani>	 great! :)
[17:57:18] <urandom>	 build uses a different base
[17:57:23] <thcipriani>	 yep
[17:57:28] <urandom>	 prep uses build
[17:58:05] <thcipriani>	 yep
[17:59:05] <urandom>	 I think what makes it confusing is either having both prep and build, or the implied relationship between them based on the hierarchy of that file
[18:00:18] <urandom>	 thcipriani: but I think I have it now, so thanks :)
[18:00:30] <thcipriani>	 "includes" vs "copies" is subtle, but in practice is crux of the idea. Ideas how to make that clearer welcome :)
[18:01:28] <ottomata>	 hey all, any details I missed on beta being in readonly mode?
[18:02:28] <thcipriani>	 urandom: cool. This is one of the few times I've attempted to explain this stuff outside of working on the documentation, so if you have ideas for how to document this more clearly they'd be welcome :)
[18:02:51] <greg-g>	 ottomata: https://phabricator.wikimedia.org/T216067 is the task to watch, really.
[18:05:53] <urandom>	 thcipriani: so one more question, is there no way to copy things to the production container arbitrarily?
[18:06:40] <urandom>	 thcipriani: are we constrained to our working directory atm?
[18:07:10] <liw>	 urandom, what would you copy from outside the working directory (= sourece tree)?
[18:07:40] <wikibugs>	 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Ottomata) Subscribing...not being able to test writes in beta is blocking some other stuff.  Thank you for working on this!
[18:07:51] <thcipriani>	 urandom: in the next release that's coming https://phabricator.wikimedia.org/T211625
[18:08:57] <urandom>	 well, SRE is saying that our config will reside as `/etc/kask/config.yaml` and that we'll use the apropos config by mounting the apropos `/etc/kask`
[18:09:40] <urandom>	 so the entrypoint in the image should be `./kask --config /etc/kask/config.yaml`, and we were going drop a default file there if possible
[18:10:14] <urandom>	 I guess helm will override the entrypoint
[18:10:27] <urandom>	 so it's more of a reasonable/documenting default
[18:10:42] <urandom>	 but it sounds like the answer is No either way (for the meantime)
[18:12:52] <thcipriani>	 hrm, I think that use-case is covered with the change at https://gerrit.wikimedia.org/r/#/c/blubber/+/485340/ (i.e., IIRC you should be able to copy a file into a container at /etc/kask/config.yaml if that's the idea), but for right now not possible.
[18:38:44] <wikibugs>	 10Release-Engineering-Team, 10docker-pkg: docker-pkg fails with "Is python3-apt installed?" - https://phabricator.wikimedia.org/T216836 (10Krinkle) If this Python module is a dependency, then it should be declared as such in setup.py so that `pip install` would pick it up, right?  I've tried installing it manu...
[18:51:22] <wikibugs>	 10Release-Engineering-Team, 10docker-pkg: docker-pkg fails with "Is python3-apt installed?" - https://phabricator.wikimedia.org/T216836 (10Krinkle) 05Open→03Invalid Thanks. Looks like it was working after all, but it just hadn't detected any changes. Thanks!
[18:51:50] <wikibugs>	 10Release-Engineering-Team, 10docker-pkg: docker-pkg fails with "Is python3-apt installed?" - https://phabricator.wikimedia.org/T216836 (10hashar) 05Invalid→03Resolved For the next error. The `python3-apt` package in the message refers to the Debian package, it is not the same as `python-apt` from pypi.  h...
[18:53:53] <wikibugs>	 (03CR) 1020after4: [C: 03+1] "Looks good to me. I haven't yet tried to run the tests locally though" [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[18:56:05] <wikibugs>	 (03CR) 1020after4: [C: 03+1] Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto)
[19:08:02] <wikibugs>	 (03CR) 1020after4: [C: 03+1] Add tests for scap.cli (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/491789 (owner: 10Giuseppe Lavagetto)
[20:00:59] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Thanks!  You will want to use 'extension-quibble', see inline comments for some guidance ;)" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/492342 (https://phabricator.wikimedia.org/T149765) (owner: 10Sophivorus)
[20:01:05] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/492342 (https://phabricator.wikimedia.org/T149765) (owner: 10Sophivorus)
[20:02:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add Extension:WikipediaExtracts [integration/config] - 10https://gerrit.wikimedia.org/r/492342 (https://phabricator.wikimedia.org/T149765) (owner: 10Sophivorus)
[20:13:56] <wikibugs>	 (03PS1) 10Krinkle: fresnel: Update docker image to install fresnel 0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/492377
[20:22:23] <wikibugs>	 (03PS1) 10Krinkle: jjb: Update fresnel job to use quibble-fresnel:0.0.28-6 (fresnel 0.2.0) [integration/config] - 10https://gerrit.wikimedia.org/r/492379
[20:26:12] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] fresnel: Update docker image to install fresnel 0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/492377 (owner: 10Krinkle)
[20:27:35] <wikibugs>	 (03Merged) 10jenkins-bot: fresnel: Update docker image to install fresnel 0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/492377 (owner: 10Krinkle)
[20:42:13] <wikibugs>	 (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Block read on refs/* to users in block-users group [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/492348 (owner: 10Paladox)
[20:42:24] <paladox>	 thanks thcipriani! :)
[20:43:14] <thcipriani>	 yw :)
[20:43:19] <wikibugs>	 10Continuous-Integration-Config, 10Discovery-Search (Current work): Setup CI for search/glent - https://phabricator.wikimedia.org/T216599 (10debt) 05Open→03Resolved
[20:59:21] <Krinkle>	 !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/492377 
[20:59:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:04:36] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban): Search broken in phabricator project tag/subscribed fields - https://phabricator.wikimedia.org/T216849 (10mmodell) a:03mmodell I'm looking into it. Thank you for the report!
[21:04:46] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Regression: Search broken in phabricator project tag/subscribed fields - https://phabricator.wikimedia.org/T216849 (10mmodell)
[21:14:53] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] jjb: Update fresnel job to use quibble-fresnel:0.0.28-6 (fresnel 0.2.0) [integration/config] - 10https://gerrit.wikimedia.org/r/492379 (owner: 10Krinkle)
[21:18:49] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Update fresnel job to use quibble-fresnel:0.0.28-6 (fresnel 0.2.0) [integration/config] - 10https://gerrit.wikimedia.org/r/492379 (owner: 10Krinkle)
[21:30:07] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Search broken in phabricator project tag/subscribed fields - https://phabricator.wikimedia.org/T216849 (10mmodell) I've just confirmed that this bug exists upstream as well... Caused by this change: https://secure.phab...
[21:44:18] <wikibugs>	 (03CR) 10Gergő Tisza: "Thanks Paladox!" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/492348 (owner: 10Paladox)
[21:44:26] <paladox>	 your welcome :)
[22:26:22] <wikibugs>	 (03PS1) 10Krinkle: Update artefact publisher to collect 'log/' instead of 'log/*' [integration/config] - 10https://gerrit.wikimedia.org/r/492401 (https://phabricator.wikimedia.org/T216622)
[22:40:23] <wikibugs>	 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team, 10Patch-For-Review: Jenkins should collect Fresnel records as build artefact - https://phabricator.wikimedia.org/T216622 (10Krinkle) p:05Triage→03High
[22:50:28] <wikibugs>	 (03PS2) 10Krinkle: Remove unused castor steps from Fresnel job [integration/config] - 10https://gerrit.wikimedia.org/r/491667
[22:58:46] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Update artefact publisher to collect 'log/' instead of 'log/*' [integration/config] - 10https://gerrit.wikimedia.org/r/492401 (https://phabricator.wikimedia.org/T216622) (owner: 10Krinkle)
[23:01:36] <wikibugs>	 (03Merged) 10jenkins-bot: Update artefact publisher to collect 'log/' instead of 'log/*' [integration/config] - 10https://gerrit.wikimedia.org/r/492401 (https://phabricator.wikimedia.org/T216622) (owner: 10Krinkle)
[23:09:42] <wikibugs>	 (03PS3) 10Krinkle: Remove unused castor steps from Fresnel job [integration/config] - 10https://gerrit.wikimedia.org/r/491667
[23:16:11] <wikibugs>	 10Phabricator (Upstream), 10MediaWiki-Vagrant, 10Upstream: Phabricator doesn't run with PHP 7.0 (default on Stretch) - https://phabricator.wikimedia.org/T185287 (10MaxSem) Since both production and Vagrant have PHP 7.2 now, this can be closed?
[23:17:46] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Remove unused castor steps from Fresnel job [integration/config] - 10https://gerrit.wikimedia.org/r/491667 (owner: 10Krinkle)
[23:20:34] <wikibugs>	 (03Merged) 10jenkins-bot: Remove unused castor steps from Fresnel job [integration/config] - 10https://gerrit.wikimedia.org/r/491667 (owner: 10Krinkle)
[23:22:18] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Query-Service: Set up Blazegraph test suite on CI - https://phabricator.wikimedia.org/T216855 (10Smalyshev)
[23:22:43] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Query-Service: Set up Blazegraph test suite on CI - https://phabricator.wikimedia.org/T216855 (10Smalyshev) p:05Triage→03Normal
[23:26:02] <wikibugs>	 10Phabricator (Upstream), 10MediaWiki-Vagrant, 10Upstream: Phabricator doesn't run with PHP 7.0 (default on Stretch) - https://phabricator.wikimedia.org/T185287 (10bd808) 05Open→03Declined >>! In T185287#4977293, @MaxSem wrote: > Since both production and Vagrant have PHP 7.2 now, this can be closed?  Ye...