[00:17:44] (03Draft1) 10Paladox: Fix node assertion to use nodejs 6 and not nodejs 4 [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) [00:17:46] (03PS2) 10Paladox: Fix node assertion to use nodejs 6 and not nodejs 4 [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) [00:21:51] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2951160 (10Liuxinyu970226) Can we also configure it for #mediawiki-l... [01:45:15] (03CR) 10Jforrester: "Why are we even asserting the node version in the first place?" [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) (owner: 10Paladox) [01:54:12] paladox whats it matter what nodejs version something has nodejs 4 code would still work in nodejs 6 correct? [02:42:17] paladox you about? [03:19:07] 10Beta-Cluster-Infrastructure: deployment-fluorine02 does not have logs - https://phabricator.wikimedia.org/T146723#2951461 (10Krenair) I've cherry-picked https://gerrit.wikimedia.org/r/#/c/313604/7 which I'm hoping will solve this [03:42:32] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2951484 (10Krenair) [03:42:36] 10Beta-Cluster-Infrastructure, 07Puppet: deployment-eventlogging03 has puppet failure due to missing class - https://phabricator.wikimedia.org/T152842#2951482 (10Krenair) 05Open>03Resolved a:03Ottomata [03:47:57] 10Beta-Cluster-Infrastructure, 03Scap3: Rewrite wmf-beta-update-databases.py as plugin - https://phabricator.wikimedia.org/T151519#2951490 (10Krenair) [04:06:05] Project selenium-MultimediaViewer ยป safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #270: 04FAILURE in 9 min 6 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/270/ [04:11:00] 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Make deployment-prep puppetmaster more similar to Production puppetmaster - https://phabricator.wikimedia.org/T146627#2666629 (10Krenair) #1 was done. [04:21:51] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: closed-labs.dblist no longer read as 'closed' tag - https://phabricator.wikimedia.org/T115584#2951534 (10Krenair) a:05ori>03Krenair [04:35:31] https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/262/console [05:51:20] (03PS2) 10Samwilson: Explicitely check for method structure before using [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/330846 (https://phabricator.wikimedia.org/T154731) [07:42:00] (03CR) 10Hashar: [C: 032] "Jobs can be mis configured to roam on different OS, the Puppet provisioning has some gotcha/surprises. The node 0.10 --> 4.0 upgrade was " [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) (owner: 10Paladox) [07:43:21] (03Merged) 10jenkins-bot: Fix node assertion to use nodejs 6 and not nodejs 4 [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) (owner: 10Paladox) [07:43:30] (03CR) 10Hashar: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/332918 (https://phabricator.wikimedia.org/T155685) (owner: 10Paladox) [08:00:16] hashar: Hey, Jenkins is broken on master of Wikibase (https://phabricator.wikimedia.org/T155600) due to failed Scribunto tests, but Scribunto extensions hasn't been touched for a while. I was wondering if some config changes can cause this [08:00:31] An example of the failure: https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/7550/console [08:01:07] Did you changed anything related to lua/scribunto recently? [08:01:13] *change [08:07:07] Amir1: do fill a task please :} Might want to link to https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/7550/testReport/ [08:07:19] Amir1: from a quick look, I guess we have changed somethings Wikibase or the CI Config [08:07:41] and the wiki sites table in MediaWiki or Wikibase changed [08:08:24] so something changed somewhere and most probably fairly recently [08:08:34] the last change that worked fine was on Jan 12 th https://gerrit.wikimedia.org/r/#/c/331909/ [08:09:04] Amir1: can copy paste that ^^^ to the task and attempt to poke #wikidata folks about it [08:09:34] hashar: Isn't this enough? https://phabricator.wikimedia.org/T155600 [08:09:55] We originally made it for another failure but it got fixed and then scribunto failed [08:10:17] Leszek is looking at the CI test failures [08:10:35] but I was wondering if something in CI config has changed yesterday [08:10:47] (the failure started from yesterday) [08:10:47] ahh reusing tasks ... :-} [08:11:08] :P [08:11:39] To be fair, title of the task is related, the reason changed only [08:12:43] yeah yeah ;D [08:13:03] pretty sure it is just about updating the scribunto test to reflect the new reality [08:14:07] oh no [08:14:08] + "//wiki.local/wiki/scribuntotitletest:Module:TestFramework", [08:14:19] "scribuntotitletest:Module:TestFramework" <-- two ':' that looks wrong ;) [08:26:02] Amir1: I have added a few more infos [08:26:05] gotta take shower / head to the bank [08:26:11] then got a conf call [08:26:15] so not sure when I can puruse [08:26:20] s/puruse/continue/ [08:26:27] hashar: I'll take a look in the mean time [08:26:29] thanks [08:26:30] gotta try to reproduce I guess [08:26:39] I would try with Wikibase + Scribunto installed [08:26:50] note that the way Wikibase extension tests are run is a bit scary [08:26:56] the context change when run under jenkins [08:27:26] so you want to set: $wgWikimediaJenkinsCI = true; [08:27:40] if that pass [08:27:48] try adding the other dependencies [08:27:56] ;-( [08:28:07] once you get a reproduction, that can be bissected [08:28:08] which is no fun [08:28:20] shower etc time [08:29:02] ah good news, bank is moved to later ;} [08:53:14] https://gerrit.wikimedia.org/r/#/q/is:merged+AND+branch:master+AND+(project:mediawiki/core+OR+project:mediawiki/extensions/BetaFeatures+OR+project:mediawiki/extensions/Capiunto+OR+project:mediawiki/extensions/CirrusSearch+OR+project:mediawiki/extensions/Cite+OR+project:mediawiki/extensions/Elastica+OR+project:mediawiki/extensions/GeoData+OR+project:mediawiki/extensions/MwEmbedSupport+OR+project:mediawiki/extensions/PdfHandler+OR+project:med [08:53:14] iawiki/extensions/Scribunto+OR+project:mediawiki/extensions/SiteMatrix+OR+project:mediawiki/extensions/TimedMediaHandler+OR+project:mediawiki/extensions/VisualEditor+OR+project:mediawiki/extensions/cldr+) [08:53:17] ARHG [08:53:18] candidates [09:30:31] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2951958 (10Marostegui) Hi, I have been doing some tests with the gerritdb to sum up all the stuff that have been discussed here... [09:36:35] !log Nuking workspaces of all mwext-testextension-hhvm-composer* jobs. Lame attempt for T155600. salt -v '*slave*' cmd.run 'rm -fR /srv/jenkins-workspace/workspace/mwext-testextension-hhvm-composer*' [09:36:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:36:41] T155600: Jenkins is broken on master - https://phabricator.wikimedia.org/T155600 [11:20:02] Amir1: I think I found the issue. SiteMatrix might be causing Scribunto tests to fail [11:21:02] hashar: awesome [11:21:15] waiting for a revert:D [12:28:34] Amir1: though I can't repro :( [12:28:43] got some random issues when trying to run tests locally [12:29:54] hashar: the strange thing is that SiteMatrix is only supposed to add a special page [12:30:09] not to mess with magic words and interwiki prefixes [12:30:47] well maybe it is BetaFeatures being at fault so :} [12:30:55] see https://phabricator.wikimedia.org/T155600#2952269 [12:32:45] away be back later [14:14:24] PHP Fatal error: Class 'getID3' not found in /home/hashar/projects/mediawiki/extensions/TimedMediaHandler/handlers/ID3Handler/ID3Handler.php on line 15 [14:14:29] * hashar whistles [14:42:21] * hashar git bisect [14:47:18] Bisecting: 2 revisions left to test after this (roughly 1 step) [14:47:29] dumb enough that I can multitask other things [15:00:25] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 06Operations, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#2952820 (10Gilles) [15:29:00] hashar hi, is that class not found still a problem? [15:34:49] paladox: nop [15:34:56] Ok [15:34:57] :) [15:34:58] I have discovered composer.local.json :] [15:35:01] ran composer update [15:35:03] oh [15:35:07] and magic happened [15:35:26] I though somehow that composer update would automagically load composer.json based on extension/skin I wfLoad [15:35:35] oh [16:23:48] is the puppet repo on deployment-puppetmaster02 somewhat / somehow behind production? e.g. https://gerrit.wikimedia.org/r/#/c/332682/ doesn't seem to be applied [16:24:42] ah yeah automatic rebase is broken, error: could not apply 8fc2a21... contint: move from /mnt to /srv [16:26:18] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894#2953153 (10hashar) I got patch for puppet.git and some of the submodules. Sent an announce to the internal ops list to gather feedback. If https://gerrit.wikimedi... [16:27:49] I've cherry-picked that patch for now, would be nice to get it fixed though, hashar perhaps ? [16:31:36] 10Continuous-Integration-Config, 06Operations, 07Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#2953176 (10hashar) [16:31:50] 10Continuous-Integration-Config, 06Operations, 07Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#2928315 (10hashar) [16:33:08] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894#2953186 (10hashar) [16:52:41] godog: eeeek [16:53:00] yeah that is a nasty patch I had on beta for quite a while and forgot / procrastrianted about it :( [16:53:18] let me fix the mess [16:56:05] !log rebased puppet master on integration and deployment-prep Trivial conflict between https://gerrit.wikimedia.org/r/#/c/312523/ and a lint change [16:56:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:56:18] godog solved :] [16:58:09] 10Deployment-Systems, 06Release-Engineering-Team, 10MediaWiki-Special-pages: All extension should link to the deployed Git revision from Special:Version - https://phabricator.wikimedia.org/T155747#2953229 (10Amire80) [17:02:20] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:04:17] hasharAway: awesome! thanks for you help :)) [17:04:55] question about gerrit and branches, we have a custom branch (es5) for working on updating CirrusSearch to use elasticsearch 5, which requires a new version of the library. To get tests running we Depends-On a patch to mediawiki/vendor containing the new code. The thing is jenkins-bot wont merge due to the depends-on. Should we branch mediawiki/vendor as well to depend on a merged patch, or manually merge? [17:06:08] (03CR) 10Umherirrender: "ArticleFeedback - ARCHIVED - https://www.mediawiki.org/wiki/Extension:ArticleFeedback" [integration/config] - 10https://gerrit.wikimedia.org/r/332890 (owner: 10Hashar) [17:07:37] ebernhardson: I think we had branch on vendor last time no? [17:07:41] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:08:09] dcausse: hmm, yes i see an es2.x branch on the remote. I suppose we can do that [17:08:17] I remember having to merge it from time to time [17:12:00] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:12:42] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:15:50] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:16:04] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:16:04] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:18:50] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:19:45] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:24:25] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:44:57] (03PS1) 10Umherirrender: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333025 [17:53:12] 10Deployment-Systems, 06Release-Engineering-Team, 10MediaWiki-Special-pages: All extension should link to the deployed Git revision from Special:Version - https://phabricator.wikimedia.org/T155747#2953229 (10Umherirrender) Works on wmflabs: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version - but tha... [17:55:14] 10Deployment-Systems, 06Release-Engineering-Team, 10MediaWiki-Special-pages: All extension should link to the deployed Git revision from Special:Version - https://phabricator.wikimedia.org/T155747#2953229 (10Reedy) See also T155624 There's also a few other bugs where we get weird edge cases, gitinfo not pro... [18:07:24] hasharAway: I know it's the end of your day... But any idea about https://gerrit.wikimedia.org/r/#/c/333029/ failing? :) [18:07:31] 18:05:26 [Symfony\Component\Console\Exception\CommandNotFoundException] [18:07:31] 18:05:26 Command "test" is not defined. [18:07:36] I can/will file a task if necessary [18:14:56] thcipriani: hi! do you have 5 minutes for a keyholder issue in deployment-tin? One of the analytics folks would need to deploy AQS but he gets "Agent admitted failure to sign using the key." [18:15:28] I checked with the commands that you gave me a while ago in https://phabricator.wikimedia.org/T116206#2251658 and everything works fine [18:16:26] (he might not be in the right group but I don't recall what is the dark magic for labs) [18:16:27] elukey: hrm, yeah lemme check. What user and service is being deployed? [18:16:46] er, what user is trying to deploy what service rather :) [18:18:03] so the user is "milimetric" and the service is aqs [18:18:28] okie doke [18:18:28] /srv/deployment/analytics/aqs/deploy [18:18:30] * thcipriani digs [18:18:34] thanks a lot! [18:22:22] elukey: milimetric give it a go now. You may have to log out and log back in. milimetric wasn't in the deploy-service group on tin so keyholder denied access to the key. [18:22:38] er deployment-tin (not tin :)) [18:22:46] can test that you can use the key with: [18:22:59] SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-aqs01.deployment-prep.eqiad.wmflabs [18:24:19] thcipriani: ah but you had to add it manually or via puppet somewhere? [18:24:59] elukey: for deployment tin I've just been adding manually since it doesn't use the same admin fanciness as production. [18:25:43] ahhhhhh [18:25:50] I didn't know it! [18:25:55] thanks a lot for the help :) [18:26:36] absotively :) [18:29:12] just sudoed as milimetric on deployment-tin, the command works perfectly now [18:29:24] * elukey awards a token of appreciation to thcipriani [18:29:49] thanks both! [18:29:58] (will try after this meeting) [18:30:42] \o/ [18:30:43] nice [18:32:03] (03CR) 10Chad: [C: 032] Add InterwikiSorting to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/332907 (https://phabricator.wikimedia.org/T150183) (owner: 10Addshore) [18:33:19] (03Merged) 10jenkins-bot: Add InterwikiSorting to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/332907 (https://phabricator.wikimedia.org/T150183) (owner: 10Addshore) [18:34:46] (03PS2) 10Chad: Add TwoColConflict to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/332906 (https://phabricator.wikimedia.org/T150184) (owner: 10Addshore) [18:36:09] (03CR) 10Chad: [C: 032] Add TwoColConflict to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/332906 (https://phabricator.wikimedia.org/T150184) (owner: 10Addshore) [18:38:06] 10Continuous-Integration-Config, 10VisualEditor, 13Patch-For-Review: Some CI jobs are failing because node is now on node 6 ("Assertion error: node version v6.9.1 does not match '^v4[.]'") - https://phabricator.wikimedia.org/T155685#2951066 (10matmarex) Fixed? [18:40:07] 10Continuous-Integration-Config, 10VisualEditor, 13Patch-For-Review: Some CI jobs are failing because node is now on node 6 ("Assertion error: node version v6.9.1 does not match '^v4[.]'") - https://phabricator.wikimedia.org/T155685#2953727 (10Jdforrester-WMF) 05Open>03Resolved a:03Jdforrester-WMF Yup,... [18:42:55] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953741 (10demon) >>! In T145885#2951958, @Marostegui wrote: > That will convert ALL tables - ie: tables with charset binary to u... [18:46:52] (03Merged) 10jenkins-bot: Add TwoColConflict to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/332906 (https://phabricator.wikimedia.org/T150184) (owner: 10Addshore) [18:48:58] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953775 (10Paladox) >>! In T145885#2951958, @Marostegui wrote: > Hi, > > I have been doing some tests with the gerritdb to sum... [18:50:42] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953779 (10demon) >>! In T145885#2953775, @Paladox wrote: > Oh, sorry didn't see your reply until now, thanks for also testing th... [18:51:49] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953780 (10Paladox) Oh, we can ignore converting the tables that were binary as well the connection is already utf8 and we haven'... [19:09:46] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953844 (10Paladox) [19:10:10] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953859 (10Paladox) [19:10:16] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2894688 (10Paladox) [19:13:12] 10Gerrit, 06Operations: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953883 (10Paladox) [19:13:46] 10Gerrit, 06Operations: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953898 (10Paladox) Im not sure if this is a #dba task? [19:15:03] 10Gerrit, 06Release-Engineering-Team: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953902 (10demon) a:03demon Not a DBA or Ops task, just me :) [19:16:34] 10Gerrit, 06Release-Engineering-Team: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953915 (10Paladox) There's a patch that needs to be merged with this. https://gerrit.wikimedia.org/r/#/c/326150/ We will also need to run https://gerrit-revi... [19:17:24] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953919 (10demon) I know. [19:18:40] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953926 (10Paladox) This patch https://gerrit.wikimedia.org/r/#/c/330455/ will need merging before taking gerrit offline. Will need to follow sql migrat... [19:18:58] (03PS1) 10Umherirrender: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333056 [19:19:46] (03CR) 10Umherirrender: "These are 10 of 20 extensions, which have enough qqq, but are not on translatewiki.net" [integration/config] - 10https://gerrit.wikimedia.org/r/333025 (owner: 10Umherirrender) [19:20:03] (03CR) 10Umherirrender: "These are the rest 10 of 20 extensions, which have enough qqq, but are not on translatewiki.net" [integration/config] - 10https://gerrit.wikimedia.org/r/333056 (owner: 10Umherirrender) [19:20:05] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953930 (10Paladox) >>! In T155766#2953919, @demon wrote: > I know. Oh sorry, when you changed the task, i was still on the old one so di... [19:23:53] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate all users username to lowercase - https://phabricator.wikimedia.org/T155766#2953941 (10Paladox) [19:23:57] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2953940 (10Paladox) [20:05:38] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2954071 (10Paladox) (i've created this patch against mysql connector, though we won't be using it, just putting it here for other... [20:06:43] hey, who should I add to this change to get it merged and get my extension deployed on the beta cluster? [20:06:43] https://gerrit.wikimedia.org/r/#/c/333086/ [20:08:21] I'll have a look after the train [20:08:50] thx! [20:13:40] Left you code review [20:21:01] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other: LinkSuggest2 test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T155773#2954140 (10Umherirrender) [20:46:32] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests, 10MediaWiki-extensions-Other: FlickrAPI test failing due to missing files located in sub repo - https://phabricator.wikimedia.org/T154847#2954251 (10Umherirrender) [20:46:35] 10Continuous-Integration-Config, 06Brickimedia, 10MediaWiki-Unit-tests, 10Refreshed: Skin Refreshed sub repo does not handled in test config - https://phabricator.wikimedia.org/T154806#2954252 (10Umherirrender) [20:46:37] 10Continuous-Integration-Config, 07TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2954245 (10Umherirrender) [20:47:29] 10Continuous-Integration-Config, 07TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2254300 (10Umherirrender) See T69216 for a task tracking some test failures (in another context) [20:55:36] (03CR) 10Hashar: [C: 032] Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333025 (owner: 10Umherirrender) [20:55:57] (03CR) 10Hashar: [C: 032] "Maybe it is worth filling a task to get them added to translatewiki.net ?" [integration/config] - 10https://gerrit.wikimedia.org/r/333025 (owner: 10Umherirrender) [20:56:05] (03CR) 10Hashar: [C: 032] Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333056 (owner: 10Umherirrender) [20:56:06] RECOVERY - Puppet run on deployment-phab02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:56:46] (03Merged) 10jenkins-bot: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333025 (owner: 10Umherirrender) [20:57:10] (03Merged) 10jenkins-bot: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333056 (owner: 10Umherirrender) [21:35:38] hello ! [21:35:38] I am trying to deploy to deployment prep [21:35:38] https://www.irccloud.com/pastebin/ciynSsTW/ [21:35:38] without any success [21:41:56] greg-g, gwicke: can any of you give me permits to deploy to deployment-aqs01 ? [21:42:44] i wonder what all is affected with tools and labs being down due to NFS [21:47:20] thcipriani: can you help nuria out ^ ? she's getting her key denied for deploy-service in beta [21:48:45] mobrovac: nuria yeah that error looks familiar. Needs to be added to the deploy-service group. Walked down to get coffee so I'm afk for a few minutes (still in line) [21:49:40] thcipriani: right, that is waht i though, you let me know when you are back [21:49:43] *what [21:51:31] Will do. It's a short walk. [21:52:00] thnx thcipriani! [22:05:50] nuria, try it now [22:06:03] Krenair: trying, thanks [22:06:24] Krenair: thanks, just sat down :) [22:06:57] Krenair: if you would be so kind to add anyone from my team? milimetric , mforns, fdans, joal , ottomata and elukey [22:07:08] Krenair: still wip [22:08:26] milimetric is on there [22:09:55] ottomata doesn't exist, what was his labs uid again? [22:10:07] otto? [22:10:30] ldapsearch -x uid=otto mail -LLL shows aotto@, so yes [22:11:43] you know this group is different on -tin to -mira [22:11:46] nuria, anyway done [22:12:32] Krenair: thank you but it failed again: [22:12:34] 22:08:24 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'analytics/aqs/deploy', '-g', 'default', 'promote', '--refresh-config'] on deployment-aqs01.deployment-prep.eqiad.wmflabs returned [70]: Failed to add the ECDSA host key for IP address '10.68.18.237' to the list of known hosts (/home/nuria/.ssh/known_hosts). [22:12:52] hm. [22:13:01] hrm, that's weird. I thought we added all those at some point. [22:13:16] relogging and retrying [22:13:19] Oh [22:13:26] Is this an NFS thing? [22:13:34] no we got rid of that [22:14:09] nuria, now? [22:14:31] still thinking [22:16:42] Krenair: no, failed again [22:17:00] nuria: could you try running: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-aqs01.deployment-prep.eqiad.wmflabs [22:17:41] nuria: I have perms to deploy, I can try if you want [22:17:54] dan should have them too thanks to Tyler [22:18:15] elukey: no need, we should all be able to do it, no worries. [22:18:32] thcipriani: [22:18:35] https://www.irccloud.com/pastebin/AwyZpL8G/ [22:18:48] Now try your deploy nuria [22:19:04] yeah it should work [22:19:31] yup, seems like the right output, looks like you can login as the deploy-service user :) [22:23:12] Krenair: it still failed (from deployment-tin) [22:23:33] Krenair: this was teh message [22:23:37] https://www.irccloud.com/pastebin/xbm7E1kh/ [22:23:59] well [22:24:27] think that one is more of a thcipriani question [22:24:41] * thcipriani digs [22:25:16] so looking at: scap deploy-log -v [22:25:19] in that same directory [22:26:32] it looks like, after the service restart, port 7232 was not reachable [22:26:36] https://phabricator.wikimedia.org/P4769 [22:27:08] thcipriani: in other words it is not a deploy error but an application error? I do not recall us using that port [22:27:10] so, deployed everything, restarted aqs, but it was not able to connect to port 7232 within the timeout limit [22:27:44] mobrovac does aqs use port 7232? [22:28:02] FWIW, scap is getting that from /srv/deployment/analytics/aqs/deploy/scap/scap.cfg it's defined as the service_port there [22:30:30] thcipriani: on aqs1004 - tcp6 0 0 :::7232 :::* LISTEN 29389/nodejs [22:30:56] checking on aqs01 [22:31:00] thcipriani what datacenter is deployment on? eqiad? [22:31:02] yeah, not seeing it there [22:31:14] (on deployment-aqs01) [22:31:41] elukey: ah, i was doing netstat to see that [22:31:53] Zppix: this is in beta (deployment-prep), so, yeap, eqiad [22:32:23] although I do see that that is the port defined in the config for aqs [22:33:06] from the logs on aqs01 it seems that node is dying each time it tries to start [22:33:18] and the logs are not there because they go to logstash [22:33:34] but we can remove that config temporarily to see what is failing [22:33:46] note that labs currently only exists in eqiad [22:34:17] (labtest doesn't count) [22:34:49] there's a beta logstash if it's configured to use it: https://logstash-beta.wmflabs.org [22:35:08] weird it seems working now [22:35:42] no that was a joke [22:36:14] heh, I was furrowing my brow at netstat pretty hard there for a second [22:36:52] elukey: maybe we can change the fact that logs are going to logstat [22:37:42] yep I was about to do it [22:37:45] elukey: where do you see the node restarts failures? [22:37:51] journalctl [22:37:54] elukey: /var/log? [22:38:10] nope, restbase doesn't log on localhost :( [22:41:27] elukey: k sudo journalctl -u aqs i guess [22:41:38] elukey: is there anything better than that to see logs? [22:41:55] not really no [22:42:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression: doc.wikimedia.org displays "403 Forbidden" for coverage sub directories - https://phabricator.wikimedia.org/T150727#2954579 (10Krinkle) >>! In T150727#2944446, @hashar wrote: > * We forgot to move the rew... [22:42:10] man... mobrovac yt? Pchelolo ? [22:42:10] the other option is to start manually the node process [22:42:25] but the issue is the one I pasted in #analytics [22:42:39] Error: Schema change, but no version increment.\n [22:44:31] question whats the access like for Beta ? [22:46:53] access to? [22:48:12] ostriches i mean like what are access restrictions like i dont quite grasp the concept of beta [22:48:35] I don't understand what you mean by access restrictions. [22:48:49] Beta is a sorta-clone of production that runs master-branch code [22:49:53] ostriches ah, ill look into it more and ill see