[00:06:17] 10Continuous-Integration-Config, 10Security: Run composer with `--dev` flag - https://phabricator.wikimedia.org/T180235#3761225 (10Reedy) [00:08:09] 10Release-Engineering-Team (Kanban), 10Diffusion, 10Phabricator (Upstream), 10Upstream: Phabricator throws Unhandled Exception ("Exception") when querying Diffusion - https://phabricator.wikimedia.org/T180457#3761237 (10mmodell) Upstreamed: https://secure.phabricator.com/D18773 [00:08:31] 10Release-Engineering-Team (Kanban), 10Diffusion, 10Phabricator (Upstream), 10Upstream: Phabricator throws Unhandled Exception ("Exception") when querying Diffusion - https://phabricator.wikimedia.org/T180457#3761240 (10mmodell) [00:10:03] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3761243 (10mmodell) >>! In T170484#3761056, @Dzahn wrote: > Having fun is also important... Indeed! [00:10:43] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] [00:12:08] lol [00:15:59] gj team [00:17:15] if the bot isn't overworked, are we all really working? [00:17:50] Current version [00:17:50] 1.29.2 – 2017-11-15 [00:17:50] 1.27.4 – 2017-11-15 [00:17:50] 1.27.4 – 2017-11-15 [00:17:54] That looks silly on the mw homepage [00:22:59] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180275#3761266 (10Reedy) [00:23:07] 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180276#3761267 (10Reedy) [00:32:35] 10Continuous-Integration-Config: Jenkins shouldn't run jobs for commits pushed with submit - https://phabricator.wikimedia.org/T180544#3761288 (10Reedy) [00:37:50] 10Release-Engineering-Team (Kanban), 10Discovery-Portal-Sprint: Create a dedicated deployment window for portal deployments - https://phabricator.wikimedia.org/T180401#3761302 (10greg) I'll let @hashar comment on any changes he thinks are needed for the deployments, but I think the process is pretty much the s... [00:40:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3761318 (10greg) >>! In T179694#3761179, @greg wrote: > Can we use a diffe... [00:42:43] so, jenkins-bot is -1-ing patches because tests aborted. Guessing the patchstorm has overwhelmed zuul. [00:42:54] if I push them through manually, will that make things worse? [00:43:04] https://gerrit.wikimedia.org/r/ [00:43:09] derp [00:43:16] https://gerrit.wikimedia.org/r/391412 [00:43:16] (03CR) 10EddieGP: [C: 04-1] "Actually that user just switched mail addresses. The commit message should not suggest that we are adding someone completely new to the wh" [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [00:44:12] ejegg: jenkins will still decide to test them at some point.. [00:44:20] But it won't block the merging if you want to CR+2 V+2 and submit [00:44:32] thanks Reedy! [00:56:47] ejegg: zuul has just been restarted [00:56:53] most of the queue is gone now :P [00:59:50] ah, cool, I bypassed it in the end [01:00:24] it was gonna be easier [01:00:31] we'll see what's failing on master and tidy up... [01:00:36] got the patches out on payments and looking fine so far [01:00:49] ...not that we actually use any wiki functionality on payments... [01:01:48] hahaha [01:01:59] everything looks like a nail! [01:02:23] SHIP IT [01:02:59] cwd actually got a long way to running DonationInterface payment forms in a minimal framework a couple years back [01:08:19] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3761407 (10EddieGP) [01:09:01] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] [01:15:32] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3761407 (10Paladox) Hi, this is a git-review tool that is owned and maintained by openstack. See https://github.com/openstack-infra/git-review You may want to contribute the patch there as this tool is not maintained o... [01:28:34] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3761447 (10EddieGP) I just noticed there already is a patch for this at https://review.openstack.org/#/c/517831/ Merging that patch will be the first step to resolve this. However, it seems nobody does releases of git-... [01:58:27] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3761407 (10Dzahn) >it seems nobody does releases of git-review nowadays How about asking the maintainer of the Debian package? Which is.. the openstack-devel list. ``` Maintainer: PKG OpenStack PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:39:59] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3761526 (10demon) 05Resolved>03Open [02:40:40] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:40:53] 10Scap, 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3432960 (10demon) p:05Triage>03Lowest Changing projects and setting priorities appropriately. [02:43:46] Project beta-scap-eqiad build #182041: 04FAILURE in 0.75 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/182041/ [02:57:00] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [03:12:51] Yippee, build fixed! [03:12:53] Project beta-scap-eqiad build #182042: 09FIXED in 19 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/182042/ [03:57:28] 10MediaWiki-Releasing: Tag MW 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180557#3761616 (10Reedy) [04:52:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:07:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:02:00] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:20:29] 10Release-Engineering-Team (Someday), 10Scap, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MBytes - https://phabricator.wikimedia.org/T145819#3761790 (10Krink... [07:20:32] 10Release-Engineering-Team (Someday), 10Deployments, 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#3761789 (10Krinkle) [07:30:45] !log Aborting jobs in 'test' pipeline for backport REL commits that are already merged meanwhile in 'submit' pipeline [07:30:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:09:47] 10Gerrit, 10Upstream: Allow searching for 'state:active', 'state:read_only', 'state:hidden' via web interface - https://phabricator.wikimedia.org/T180297#3761853 (10MarcoAurelio) Thank you @Paladox It might make sense to allow for state:hidden as well but for those with the relevant permissions :) [08:47:21] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch - https://phabricator.wikimedia.org/T180524#3761888 (10Joe) @dduvall the reason this is happening right now is that stretch doesn't have a package for npm and I didn't g... [08:48:17] no_justification: having different versions in different places will start to bit us now (in terms of wikibase / wikidata) [08:48:43] there is no reason we can't have the build auto branch each week now! I'll be sure to turn that on today and maybe bring Wikidata up to date [09:04:19] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3761407 (10Legoktm) If I can plug one of my own tools: . It's pretty simple to use, but has a different model of development as it relies on detached HEADs and not branches. [09:13:54] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3752019 (10Kghbln) @Reedy Thanks a lot for making the releases! This is very much appreciated. The tags for the newly released versions are still missing. Is T180557 a blocker here? [09:15:30] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch - https://phabricator.wikimedia.org/T180524#3761976 (10MoritzMuehlenhoff) Current npm releases are not packaged in Debian since the list of dependencies exploded. Effort... [09:32:44] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:53:16] (03PS1) 10Addshore: Branch Wikidata extension repo from master each week [tools/release] - 10https://gerrit.wikimedia.org/r/391524 (https://phabricator.wikimedia.org/T180539) [09:55:33] !log created wmf/1.31.0-wmf.8 branch of Wikidata extension repo T180539 [09:55:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:55:39] T180539: wmf.8 still on old Wikidata branch - https://phabricator.wikimedia.org/T180539 [10:12:44] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:13:22] Reedy: around? :) [10:41:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Run Cucumber+Selenium+Node.js in CI - https://phabricator.wikimedia.org/T179190#3716147 (10zeljkofilipin) a:03zeljkofilipin [10:48:18] 10Gerrit, 10Upstream: "git review -d" is broken - https://phabricator.wikimedia.org/T180548#3762304 (10Paladox) Whilst at it should we fix that issue with having a base url for https clones? Base url as in /r as many users are having issues with it trying /changes/ instead of /r/changes/? [11:01:00] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Run Cucumber+Selenium+Node.js in CI - https://phabricator.wikimedia.org/T179190#3762336 (10zeljkofilipin) [11:08:43] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3762366 (10Aklapper) [11:12:04] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3762373 (10zeljkofil... [11:25:41] (03CR) 10WMDE-leszek: [C: 031] Branch Wikidata extension repo from master each week [tools/release] - 10https://gerrit.wikimedia.org/r/391524 (https://phabricator.wikimedia.org/T180539) (owner: 10Addshore) [11:36:30] (03CR) 10Ladsgroup: [C: 032] Branch Wikidata extension repo from master each week [tools/release] - 10https://gerrit.wikimedia.org/r/391524 (https://phabricator.wikimedia.org/T180539) (owner: 10Addshore) [11:37:00] (03Merged) 10jenkins-bot: Branch Wikidata extension repo from master each week [tools/release] - 10https://gerrit.wikimedia.org/r/391524 (https://phabricator.wikimedia.org/T180539) (owner: 10Addshore) [12:23:09] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:28:12] 10Gerrit, 10Operations, 10Readers-Web-Backlog, 10Patch-For-Review, and 2 others: [subtask] Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3762638 (10phuedx) > In T178189#3740029, @akosiaris wrote: >> In T178189#3740805, @phuedx wrote: >> One thing that we (Readers Web)... [12:29:32] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3762643 (10zeljkofilipin) [12:30:52] 10Release-Engineering-Team (Watching / External), 10Electron-PDFs, 10Operations, 10Proton, and 3 others: [subtask] How should we get Chromium for use in puppeteer? - https://phabricator.wikimedia.org/T178570#3762650 (10phuedx) [12:32:08] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3762651 (10zeljkofilipin) [12:34:32] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3762657 (10zeljkofil... [12:34:33] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3762656 (10zeljkofilipin) [12:34:35] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3762658 (10zeljkofilipin) [12:34:56] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3247401 (10zeljkofilipin) [12:34:58] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3762661 (10zeljkofilipin) [12:35:01] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Cucumber+Selenium+Node.js in CI - https://phabricator.wikimedia.org/T179190#3762659 (10zeljkofilipin) [12:35:23] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3247408 (10zeljkofilipin) [12:35:26] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125#3762662 (10zeljkofilipin) [12:35:30] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3748673 (10zeljkofilipin) [12:36:48] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3762666 (10zeljkofilipin) [12:36:51] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3750228 (10zeljkofilipin) [12:36:53] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3762665 (10zeljkofilipin) [12:38:36] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3762669 (10zeljkofilipin) [12:58:26] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3762754 (10Dzahn) I wonder where that comes from since i have used git review for many many years and don't remember ever having that issue you are referring too, Paladox. [13:12:00] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3762819 (10Paladox) @Dzahn there’s a task around some where but there have been many tasks. Even been asked privately about it. I can look for the task later but may be harde... [13:27:02] Is eswiki in group1? [13:27:07] greg-g: ^ [13:30:54] Or anyone from releng^^ [13:32:46] To give a bit of context (and help you to fix something), https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap and versions after say it is in group 1 but not https://tools.wmflabs.org/versions/ [13:34:44] kart_: no it's group2 according to expanddblist group2 [13:37:25] Trizek: as I guessed ^ [13:38:29] Why has it been added to that Mediawiki page then? It is a bit disturbing. [13:38:52] And it is the most public page: you have a list from Tech News to that page every week. :D [13:39:14] Trizek: where do you see eswiki being part of group1? [13:39:41] dcausse: https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap and onwards.. [13:39:43] https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap ad all roadmaps pages created after that. [13:41:12] The page has been setup by Florianschmidtwelzow, with Spanish on group 1. https://www.mediawiki.org/w/index.php?title=MediaWiki_1.28/Roadmap&oldid=2101505 [13:41:16] Trizek: it's possible that eswiki moved from group1 to group2, last edit on this page is nov 2016? [13:41:41] Yes, 2 November 2016‎ [13:42:23] Bu Dut the mistake has been replicated since then: https://www.mediawiki.org/w/index.php?title=MediaWiki_1.29/Roadmap&oldid=2256646 [13:43:38] yes looks like a mistake, I can't find anything mentioning that eswiki moved from group1 to group2 after nov 2016 [13:44:28] hte only wikipedias that are explicitly on group1 are catalan and hebrew [13:47:17] right [13:59:26] I think the cause is here: https://www.mediawiki.org/w/index.php?title=Template:WMFReleaseTableRow&diff=next&oldid=2436257 [14:00:03] An IP has added Spanish Wikipedia. Why? Maybe to have the deployment first? :) [14:01:48] I haven't checked how the page was built. You know what happen when a template is changed... [14:26:41] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3763077 (10EBernhard... [15:01:47] (03PS2) 10Thcipriani: Convert ci-jessie and ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/388447 (owner: 10Giuseppe Lavagetto) [15:17:13] (03PS3) 10Giuseppe Lavagetto: Convert ci-jessie and ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/388447 [15:17:17] (03PS2) 10Giuseppe Lavagetto: Convert operations-puppet to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388448 [15:17:19] (03PS2) 10Giuseppe Lavagetto: Convert tox to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388449 [15:17:21] (03PS2) 10Giuseppe Lavagetto: Convert npm, npm-test to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388450 [15:17:24] (03PS2) 10Giuseppe Lavagetto: Convert php to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388451 [15:17:25] (03PS2) 10Giuseppe Lavagetto: Convert all composer images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388452 [15:17:27] (03PS1) 10Giuseppe Lavagetto: Convert mediawiki-* images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/391564 [15:33:13] (03CR) 10Thcipriani: [C: 032] "Will build this image manually on contint1001 initially. Other contint-admins should be able to do likewise. Will automate via jenkins/fab" [integration/config] - 10https://gerrit.wikimedia.org/r/388447 (owner: 10Giuseppe Lavagetto) [15:34:18] (03Merged) 10jenkins-bot: Convert ci-jessie and ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/388447 (owner: 10Giuseppe Lavagetto) [15:43:37] (03PS3) 10Giuseppe Lavagetto: Convert tox to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388449 [15:43:39] (03PS3) 10Giuseppe Lavagetto: Convert npm, npm-test to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388450 [15:43:41] (03PS3) 10Giuseppe Lavagetto: Convert php to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388451 [15:43:43] (03PS3) 10Giuseppe Lavagetto: Convert all composer images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388452 [15:43:45] (03PS2) 10Giuseppe Lavagetto: Convert mediawiki-* images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/391564 [15:56:17] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3763316 (10zeljkofilipin) a:03zeljkofilipin [15:57:14] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3585179 (10zeljkofilipin) Test job: [[ https://integration.wikimedia.org/ci/job/selenium-CirrusSe... [16:02:27] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3761407 (10demon) >>! In T180548#3761947, @Legoktm wrote: > If I can plug one of my own tools: . It's pretty simple to use, but has a different... [16:04:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3763341 (10Jdforrester-WMF) [16:11:07] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:19:10] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3763382 (10Dzahn) >>! In T180548#3761447, @EddieGP wrote: > I just noticed there already is a patch for this at https://review.openstack.org/#/c/517831/ > Merging that patch w... [16:23:48] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:24:48] (03CR) 10Thcipriani: [C: 032] Convert operations-puppet to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388448 (owner: 10Giuseppe Lavagetto) [16:26:34] (03Merged) 10jenkins-bot: Convert operations-puppet to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/388448 (owner: 10Giuseppe Lavagetto) [16:31:34] no_justification lol at your answer for git-review :) [16:31:40] the docs should be updated [16:31:46] to use plain git [16:32:16] Meh, I stopped caring years ago [16:32:22] If people wanna use a shitty tool, that's on them [16:32:43] yep. /me dosen't use the tool, you can copy the command from gerrit :) [16:32:54] Trizek: kart_ looks like a mistake with something a volunteer was maintaining wrt to deployments. It's not on the only page I maintain: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20171115T2000 [16:33:24] Hi no_justification! :) [16:33:36] paladox: indeed, i also just use git [16:33:37] Trizek: kart_ those "roadmap" pages are not maintained by us [16:33:43] :) [16:33:52] anyways it is broken with gerrit 2.14 [16:34:05] a fix was submitted upstream but they have not done a release since 2015 [16:37:43] (03PS1) 10Thcipriani: Remove now unused prebuild.sh [integration/config] - 10https://gerrit.wikimedia.org/r/391574 [16:37:45] (03PS1) 10Thcipriani: Match docker-pkg-config.yaml to contint1001 [integration/config] - 10https://gerrit.wikimedia.org/r/391575 [16:37:47] (03PS1) 10Thcipriani: Update example-run to match docker-pkg output [integration/config] - 10https://gerrit.wikimedia.org/r/391576 [16:41:42] (03CR) 10Thcipriani: [C: 032] Remove now unused prebuild.sh [integration/config] - 10https://gerrit.wikimedia.org/r/391574 (owner: 10Thcipriani) [16:41:56] (03CR) 10Thcipriani: [C: 032] Match docker-pkg-config.yaml to contint1001 [integration/config] - 10https://gerrit.wikimedia.org/r/391575 (owner: 10Thcipriani) [16:43:35] (03Merged) 10jenkins-bot: Remove now unused prebuild.sh [integration/config] - 10https://gerrit.wikimedia.org/r/391574 (owner: 10Thcipriani) [16:43:37] (03Merged) 10jenkins-bot: Match docker-pkg-config.yaml to contint1001 [integration/config] - 10https://gerrit.wikimedia.org/r/391575 (owner: 10Thcipriani) [16:47:05] (03CR) 10Thcipriani: [C: 032] Update example-run to match docker-pkg output [integration/config] - 10https://gerrit.wikimedia.org/r/391576 (owner: 10Thcipriani) [16:48:24] (03Merged) 10jenkins-bot: Update example-run to match docker-pkg output [integration/config] - 10https://gerrit.wikimedia.org/r/391576 (owner: 10Thcipriani) [16:48:43] addshore: G'morning [16:48:49] I saw your ping when I woke up [16:48:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [16:49:31] no_justification: so we tried to get wikidata on .8, but ran into https://phabricator.wikimedia.org/T180605 :/ [16:58:55] addshore: Looks like Daniel has a hacky fix for now? [17:01:21] no_justification im going to test logstash now for gerrit :). (Currently installing) [17:01:31] no_justification: and i checked the ferm rules [17:02:01] port 4560 is allowed from cobalt/gerrit, it uses $DOMAIN_NETWORKS, which includes the public range [17:02:14] or was it another rule that was needed? [17:02:55] I think that was it. Originally we hadn't even done ferm rules :) [17:04:41] no_justification: yup [17:05:03] paladox: ^ are we really using the hostname in the config though? [17:05:08] wasnt't there a service name [17:05:23] it's crossed out [17:05:26] once uncrossed we will [17:05:43] https://github.com/wikimedia/puppet/blob/3112c8b002996e228474361b9cd54755718e04ab/hieradata/role/common/gerrit/server.yaml [17:05:45] no_justification: if its okay with you id like to merge that, update the .8 branch for wikidata, and roll that out to testwikidata before the train today? [17:06:31] paladox: no_justification: [17:06:31] logstash.svc.eqiad.wmnet has address 10.2.2.36 [17:06:40] that seems better than using logstash1002 [17:06:40] ah [17:06:43] yeh [17:07:58] 10Gerrit, 10Patch-For-Review: 404 downloading any changes with https remote url: "The requested URL /changes/ was not found" - https://phabricator.wikimedia.org/T154760#3763535 (10Paladox) Hi, im wondering if some users will be able to test https://review.openstack.org/#/c/478325/ please? It is a fix for this... [17:09:53] 10Gerrit, 10Upstream: Git 2.15: "git review -d" is broken since "-set-upstream" got removed - https://phabricator.wikimedia.org/T180548#3763546 (10Paladox) Upstream are planning on releasing a new version soon :). They may also include a fix for this T154760 [17:12:12] addshore: Sounds like a plan [17:12:35] paladox, mutante: Yes, lets use the service address [17:12:41] ok :) [17:13:11] addshore: Assuming we can't get it fixed by the train window, should the train continue just holding back Wikidata for now? [17:14:35] I'd have to do some checking, it should probably be held back, as, for example, all of the composer package version in mediawiki-vendor (which is the code used in prod) have version matched to Wikidata .8 not .6 [17:15:00] I dont think anything major has happened in the packages, but it is just a recipe for disaster if things start diverging like this [17:15:32] If we get .8 done today / in this train I should be able to totally finish killing the build next monday tuesday [17:18:10] (03PS4) 10Zoranzoki21: Add TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 [17:18:35] (03PS5) 10Zoranzoki21: Added secondary mail of user TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 [17:18:39] (03PS6) 10Zoranzoki21: Added secondary mail of user TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 [17:19:22] (03CR) 10Dzahn: [C: 031] Added secondary mail of user TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [17:19:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3763562 (10greg) >>! In T179694#3761318, @greg wrote: > Really, the accoun... [17:24:00] (03CR) 10EddieGP: [C: 031] "Thanks, that looks good :)" [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [17:25:02] (03CR) 10Zoranzoki21: "> Thanks, that looks good :)" [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [17:25:47] meh, I will have to backport it in wikibase too i guess [17:26:26] i cant wait for this to all die [17:29:45] !log updating docker-pkg dockerfiles on contint1001 for https://gerrit.wikimedia.org/r/#/c/388448/ [17:29:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:32] (03PS1) 10Arturo Borrero Gonzalez: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 [17:31:14] (03CR) 10jerkins-bot: [V: 04-1] commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 (owner: 10Arturo Borrero Gonzalez) [17:33:44] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:35:33] no_justification: are you okay for me to use the start of the train window? :) [17:35:58] / end of swat & start of train [17:37:18] addshore: Go for itttttt [17:37:19] :) [17:38:27] and as there is no train next week, I was gonna schedule some largish slots on monday and tuesday to try and wrap this all up :) [17:38:38] greg-g: ^^ [17:40:33] (03PS2) 10Arturo Borrero Gonzalez: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 [17:41:13] (03CR) 10jerkins-bot: [V: 04-1] commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 (owner: 10Arturo Borrero Gonzalez) [17:41:19] addshore: if it's cool with no_justification I'm cool with it [17:41:24] (03PS1) 10Thcipriani: New fab task for rebuilding docker-pkg images [integration/config] - 10https://gerrit.wikimedia.org/r/391592 [17:41:59] Of course I'm cool with it :) [17:42:56] wheee :) [17:44:51] (03PS3) 10Arturo Borrero Gonzalez: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 [17:45:37] (03CR) 10jerkins-bot: [V: 04-1] commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 (owner: 10Arturo Borrero Gonzalez) [17:47:10] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch - https://phabricator.wikimedia.org/T180524#3763641 (10dduvall) Thanks for the update @joe and @MoritzMuehlenhoff! What do you think about maintaining separate tags bas... [17:54:21] (03PS4) 10Arturo Borrero Gonzalez: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 [18:04:09] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:08:42] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:55] bd808: I remember when originally developing commit-message-validator there was a web page that listed a bunch of standard git footers that kernel/etc. used, and I can't find it right now [18:16:17] (03CR) 10Legoktm: [C: 04-1] "This looks fine except for the Depends-On change which should stay uppercase since that's how zuul stylizes them: 10Release-Engineering-Team (Watching / External), 10Electron-PDFs, 10Operations, 10Proton, and 3 others: [subtask] How should we get Chromium for use in puppeteer? - https://phabricator.wikimedia.org/T178570#3763739 (10bmansurov) a:05bmansurov>03None [18:17:26] 10Release-Engineering-Team (Kanban), 10User-greg: Get firm offsite budget for Lani - https://phabricator.wikimedia.org/T180619#3763741 (10greg) [18:20:21] 10Release-Engineering-Team (Watching / External), 10Electron-PDFs, 10Operations, 10Proton, and 4 others: How should we get Chromium for use in puppeteer? - https://phabricator.wikimedia.org/T178570#3763759 (10bmansurov) [18:21:07] 10Gerrit, 10Operations, 10Readers-Web-Backlog, 10Patch-For-Review, and 2 others: [subtask] Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3763761 (10bmansurov) a:05bmansurov>03None [18:22:08] 10Gerrit, 10Operations, 10Readers-Web-Backlog, 10Patch-For-Review, and 3 others: [spike] Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3763762 (10phuedx) [18:22:17] 10Gerrit, 10Operations, 10Readers-Web-Backlog, 10Patch-For-Review, and 3 others: [spike] Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683850 (10phuedx) a:03phuedx [18:22:59] 10Release-Engineering-Team (Kanban), 10User-greg: Get sunsetting doc feedback from CE - https://phabricator.wikimedia.org/T180620#3763766 (10greg) [18:40:25] (03PS5) 10Arturo Borrero Gonzalez: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 [18:58:42] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch - https://phabricator.wikimedia.org/T180524#3763858 (10Joe) @MoritzMuehlenhoff did you ever took a look at the deb packages that are officially distributed by node? The... [19:02:35] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Use dep to manage vendor dependencies for blubber - https://phabricator.wikimedia.org/T180530#3763861 (10dduvall) [19:05:26] paladox: Got lfs plugin built and uploaded: https://gerrit.wikimedia.org/r/#/c/391625/ [19:05:47] Then we can test it on a dummy repo [19:05:59] So we'll be ready at that point if/when a repo needs it :) [19:19:48] 10Release-Engineering-Team (Kanban), 10User-greg: Create "what we need" doc - https://phabricator.wikimedia.org/T180625#3763965 (10greg) [19:22:29] (03CR) 10Legoktm: [C: 032] "Thanks!" [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 (owner: 10Arturo Borrero Gonzalez) [19:23:15] (03Merged) 10jenkins-bot: commit_message_validator: homogeinize case in footers [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/391590 (owner: 10Arturo Borrero Gonzalez) [19:23:32] Yippee, build fixed! [19:23:33] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #199: 09FIXED in 34 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/199/ [19:26:30] 10Scap: Support git-lfs - https://phabricator.wikimedia.org/T180627#3764014 (10demon) [19:29:10] no_justification :) [19:29:13] already tested [19:29:15] for 2.14 [19:29:25] https://gerrit.wikimedia.org/r/#/c/363734/21 [19:29:37] it's live at https://gerrit.git.wmflabs.org/r/ [19:30:33] 10Scap, 10Operations: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#3764041 (10demon) [19:32:11] no_justification i have a patch on gerrit.wm.org ready for when we install the plugin [19:32:27] https://gerrit.wikimedia.org/r/#/c/385239/ [19:32:31] 10Release-Engineering-Team (Kanban), 10Diffusion, 10Phabricator (Upstream), 10Upstream: Phabricator throws Unhandled Exception ("Exception") when querying Diffusion - https://phabricator.wikimedia.org/T180457#3764059 (10mmodell) This is resolved by {rPHAB62d1fa986} which should be deployed tonight. [19:38:51] 10Scap, 10Operations: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#3764068 (10demon) @thcipriani Noticed that it appears it's been [[ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792075 | added ]] to [[ https://packages.debian.org/source/buster/gi... [19:40:27] paladox: Yeah, I saw your change for enabling it for the chromium-render repo. But it looks like they're just gonna use native debian packages. [19:40:35] So we can just test it on some dummy repo [19:40:36] yep [19:40:40] yeh [19:41:01] my patch can be changed for a different repo when the plugin is installed :) [19:42:09] Just deployed the plugin, just gotta wait the 10m for the plugin refresh to happen [19:42:17] Sometime between now & 10m, at least :) [19:42:29] yeh :) [19:42:34] And bam, done [19:42:35] lfs 210fd35 Enabled [19:42:36] it should be imediate i think [19:42:38] heh :) [19:42:47] time to test. [19:42:59] Ah, Default is 1 minute. [19:43:00] :) [19:43:06] wonder which repo should we test :). and yeh :) [19:43:19] Maybe just gerrit-ping? [19:43:22] yeh [19:43:24] will do that [19:43:28] * paladox updates patch :) [19:44:09] done [19:44:41] on the project config page [19:44:45] we should see something like [19:44:47] LFS Options [19:44:47] Enabled: true [19:44:47] Max Object Size: 524288000 [19:44:47] Read Only: false [19:44:48] Backend: gerrit-fs [19:44:56] when the change is merged [19:46:26] Merged. [19:46:31] yay [19:46:32] https://gerrit.wikimedia.org/r/#/admin/projects/test/gerrit-ping [19:46:34] works :) [19:46:37] afk for a minute, walking the dog [19:46:43] ok :) [19:46:51] And wheeeee :D [19:46:55] :) [19:47:53] 10Release-Engineering-Team (Kanban), 10Diffusion, 10Phabricator (Upstream), 10Upstream: Phabricator throws Unhandled Exception ("Exception") when querying Diffusion - https://phabricator.wikimedia.org/T180457#3764081 (10MarcoAurelio) Thank you for the quick response and fix. [19:49:00] wheee no_justification it works when git cloning (though i do get a failed message as i doint have git-lfs-authentication installed) [19:49:19] https://phabricator.wikimedia.org/P6322 [19:50:45] ah [19:50:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [19:51:03] maybe because i have to set url in .git per docs. I wish they supported not setting this. [19:51:21] 10Continuous-Integration-Config: Jenkins isn't showing mediawiki-core-php70-phan-docker failure output - https://phabricator.wikimedia.org/T180630#3764103 (10Anomie) [19:51:32] oh only needed if you use ssh [19:55:39] https works [19:55:52] ssh should work as long as you set [19:55:57] lfs.url in .git/config [19:57:11] https://gerrit.wikimedia.org/r/391629 [19:58:19] hmm i did not run the git-lfs command to upload ^^ [19:59:54] hmm it hangs when i git lfs push origin master [20:00:40] Not seeing anything in error_log for it [20:01:40] i wonder if it's http having the problem, switching to ssh now. [20:03:54] still hangs [20:04:16] im not git lfs pushing anything large, just a simple index.php file with echo "testing lfs"; [20:10:07] yay [20:10:15] no_justifications figured it out [20:10:20] no_justification [20:10:36] https://phabricator.wikimedia.org/P6323 [20:11:01] and hangs again [20:13:41] aha [20:13:42] 20:13:33.063204 trace git-lfs: Filled credentials for https://paladox@gerrit.wikimedia.org/r/a/test/gerrit-ping/info/lfs [20:13:42] 20:13:33.063268 trace git-lfs: HTTP: POST https://paladox@gerrit.wikimedia.org/r/a/test/gerrit-ping/info/lfs/locks/verify [20:13:42] 20:13:33.412604 trace git-lfs: HTTP: 401 [20:13:58] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2017-10-24 (1.31.0-wmf.5)), 10Patch-For-Review: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3764492 (10Jdrewniak... [20:14:01] Got a stacktrace this time [20:14:17] Git LFS: (0 of 1 files) 0 B / 28 B [20:14:17] batch response: No LFS plugin is configured to handle LFS requests. [20:14:17] error: failed to push some refs to 'ssh://paladox@gerrit.wikimedia.org:29418/test/gerrit-ping' [20:14:29] https://phabricator.wikimedia.org/P6324 [20:14:29] removing /a/ got me to that error now [20:14:46] bug [20:15:04] Oh, duh [20:15:14] We have to set lfs.plugin = lfs in gerrit.config [20:15:17] We set the other bits [20:15:24] Can anyone help me understand how parserTests.txt modifies other tests? [20:15:26] But that kinda needed to wait for the plugin to...exist [20:15:27] Hehe [20:15:39] This failure is really confusing: [20:15:46] https://integration.wikimedia.org/ci/job/mediawiki-phpunit-php55-jessie/1011/console [20:16:00] if you look at the source of LanguageSrTest::testConversionToLatin [20:16:02] https://groups.google.com/forum/#!topic/repo-discuss/RDxiyotcsog [20:16:23] There's nothing about that XSS [20:26:23] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #13: 04FAILURE in 39 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/13/ [20:26:38] no_justification i've got logstash installed http://gerrit-logstash.wmflabs.org :) [20:26:39] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3764566 (10Reedy) >>! In T180272#3761971, @Kghbln wrote: > @Reedy Thanks a lot for making the releases! This is very much appreciated. The tags for the newly released versions are still missing... [20:40:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:43:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3764629 (10Ciencia_Al_Poder) [20:46:57] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3764656 (10Kghbln) > Kind of. I made the task as the last thing I did last night as a reminder/notice to other people that it was a known TODO :) - Wed, Nov 15, 3:57 AM Oops, so I was in bed e... [20:47:17] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3764659 (10Addshore) [20:47:20] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Platform-Team, 10Wikidata: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? - https://phabricator.wikimedia.org/T179663#3764657 (10Addshore) 05Resolved>03Open I'm going to... [20:53:38] hey, release engineers with +2 on integration/config, wanna merge https://gerrit.wikimedia.org/r/#/c/391079/ looks good and harmless but i dont have +2 on that [20:54:01] "Added secondary mail of user TerraCodes to CI whitelist. [20:54:14] (03CR) 10Thcipriani: [C: 032] Added secondary mail of user TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [20:54:19] thanks :) [20:54:24] np :) [20:54:38] good looking-out [20:56:01] (03Merged) 10jenkins-bot: Added secondary mail of user TerraCodes to CI whitelist. [integration/config] - 10https://gerrit.wikimedia.org/r/391079 (owner: 10Zoranzoki21) [20:56:28] rumours are it needs a deployment/restart step [20:56:47] yeah, zuul reload [20:56:56] I'll do that [20:57:04] 'k :) thx [20:58:20] greg-g: no_justification I added some large ish slots on monday and tuesday for me to finish everything up, as well as a slot to do some stuff tomorrow [20:59:22] no_justification: Perhaps we could do https://gerrit.wikimedia.org/r/#/c/391251/ today or tomorrow? beta seems like it was fine with the change! [21:00:08] I think I am off for now though! [21:01:25] (03PS1) 10Thcipriani: Remove comment from big email regex [integration/config] - 10https://gerrit.wikimedia.org/r/391652 [21:03:13] (03CR) 10Thcipriani: [C: 032] Remove comment from big email regex [integration/config] - 10https://gerrit.wikimedia.org/r/391652 (owner: 10Thcipriani) [21:04:29] (03Merged) 10jenkins-bot: Remove comment from big email regex [integration/config] - 10https://gerrit.wikimedia.org/r/391652 (owner: 10Thcipriani) [21:04:38] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3764733 (10Reedy) Haha. {T180275} and {T180276} were release-note-y stuff The plan wasn't for it to be 24 hours later, but unfortunately, these things happen {T180557} should probably be a pa... [21:05:13] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/391079/6 [21:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:09:29] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3764784 (10Addshore) [21:12:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3764800 (10jmatazzoni) [21:24:01] aha [21:24:10] no_justification did you build the lfs plugin with 2.14? [21:24:14] it wont work with 2.13 [21:24:18] as it uses locks [21:24:26] for example it shows [21:24:31] 21:20:58.814627 trace git-lfs: HTTP: POST https://paladox@gerrit.wikimedia.org/r/a/test/gerrit-ping.git/info/lfs/locks/verify [21:24:36] 2.13 [21:24:39] which the url dosent work for me [21:24:46] Since we run 2.13 ;-) [21:24:49] but this does [21:25:02] https://gerrit.git.wmflabs.org/r/a/test.git/info/lfs/locks/verify [21:26:35] 21:20:57.789853 trace git-lfs: creds: git credential fill ("https", "gerrit.wikimedia.org", "r/a/test/gerrit-ping") [21:26:36] 21:20:57.830109 trace git-lfs: Filled credentials for https://paladox@gerrit.wikimedia.org/r/a/test/gerrit-ping [21:27:27] 21:27:05.464491 trace git-lfs: HTTP: {"message":"User anonymous is not authorized to perform upload operation"} [21:35:08] no_justification works for me on my test site [21:35:14] https://gerrit.git.wmflabs.org/r/ [21:35:21] though only difference is it is running 2.14. [21:35:37] To be honest, I wanna start planning the 2.14 upgrade anyway [21:35:39] https://gerrit.git.wmflabs.org/r/#/c/23/1/index.php [21:35:44] yeh [21:35:49] i have patches for zuul [21:36:26] i have tested the zuul patches in my test install. The patches should be good for now and wont require hashar to do a full zuul upgrade :); [21:43:28] no_justification does lfs work for you as an admin? [21:43:58] I haven't tested [21:44:05] ok [21:44:56] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3764852 (10Reedy) [21:44:58] 10MediaWiki-Releasing: Tag MW 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180557#3764849 (10Reedy) 05Open>03Resolved a:03Reedy Tags created [21:45:36] 10MediaWiki-Releasing: Tag MW 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180557#3761616 (10Kghbln) >>! In T180557#3764849, @Reedy wrote: > Tags created Cool. Thanks a lot! [21:50:03] hmm i wonder do we need to set [21:50:03] auth] [21:50:03] gitBasicAuth = true [21:50:10] though if it works for me without that [22:00:17] /srv/gerrit/plugins/lfs/ [22:00:44] ill wait for the upgrade to 2.14 as it seems to work for me unless being an admin matters for lfs. [22:16:05] no_justification im going to flip the switch on gerrit's side to test logstash now heh [22:16:17] after messing with forward ports for about an hour trying to get it to work [22:17:29] :) [22:24:19] no_justification we will need load the log4j config. Do i have to do -Dlog4j.configuration=file:///home/gerrit/site/etc/log4j.properties some how in the systemd script? [22:26:37] Yeahhh [22:27:58] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180272#3764960 (10Bawolff) [22:28:00] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.4/1.28.3/1.29.2 - https://phabricator.wikimedia.org/T180275#3764958 (10Bawolff) 05Open>03Resolved a:03Bawolff [22:32:23] addshore: Warning: preg_match() expects parameter 2 to be string, array given in /srv/mediawiki/php-1.31.0-wmf.8/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/Helper/SparqlHelper.php on line 462 :( [22:33:38] no_justification Nov 15 22:31:29 gerrit-test3 java[8414]: log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed) [22:35:56] :\ [22:37:24] https://github.com/logstash-plugins/logstash-input-log4j/issues/13 [22:38:02] apparently using tcp worked around the problem. But i am wondering how they managed to use tcp [22:44:46] no_justification it seems to be because they are using log4j 2.x [22:44:51] whereas gerrit is using 1.2 [22:45:04] though it's suposidly fixed in https://github.com/elastic/logstash/commit/8dfefad58a1d5257016b1225994abfed2306d8ca [22:46:19] hmm which is supposidly fixed in 5.2.0 [22:46:53] paladox: Tbh, sending it over the log4j socket appender was always kind of kludgy. I wonder if there's a better option [22:47:03] oh yrh [22:47:04] yeh [22:47:11] im looking into a better option :) [22:47:18] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:47:23] (03PS1) 10Chad: Use --jobs in git submodule update jobs [integration/config] - 10https://gerrit.wikimedia.org/r/391712 [22:48:12] [2017-11-15T23:13:25,269][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#, :backtrace=>["org/jruby/RubyIO.java:3705:in `select'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:93:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:56:in `run'", "/usr/share/logstash/logstash-co [22:48:12] re/lib/logstash/pipeline.rb:425:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `start_input'"]} [22:48:13] re/lib/logstash/pipeline.rb:425:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `start_input'"]} [22:50:18] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:51:06] (03CR) 10Chad: "Wrote this because phpcs-docker was really slow for wmf.* branches. I'm *pretty sure* ci-src-setup-simple is the place I want to fix it." [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [22:54:59] no_justification i found the error [22:55:02] by setting debug [22:55:03] https://phabricator.wikimedia.org/P6325 [22:55:04] (03CR) 10Legoktm: "phpcs-docker shouldn't be checking out submodules, that's a bug :/" [integration/config] - 10https://gerrit.wikimedia.org/r/391712 (owner: 10Chad) [22:55:07] it's an internal error [22:55:13] possibly a bug in logstash [23:02:19] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:10:18] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:12:35] no_justification spoke with ebernhardson and he suggested to me we use gelf so im going to test that :). [23:12:47] Yeah, that's what I was trying to remember [23:12:48] gelf [23:13:16] you sent something not-json to an endpoint that decodes json :P [23:14:42] heh :) [23:18:08] paladox: how do you award badges to people in phab ?:) [23:18:21] mutante you have to be the admin i think. [23:18:22] "User has not been awarded any badges." :) [23:18:27] or be allowed to award badges [23:18:51] ok [23:37:01] 10Release-Engineering-Team (Kanban), 10User-greg: Get firm offsite budget for Lani - https://phabricator.wikimedia.org/T180619#3763741 (10greg) 05Open>03Resolved