[00:05:22] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:13:10] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2856022 (10RobH) [00:13:20] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856023 (10Smalyshev) [00:14:18] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2851729 (10RobH) [00:15:51] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856023 (10demon) I'm pretty sure this is actually because of gerrit restarting automatically when it shouldn't. Working on it. [00:16:07] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856044 (10demon) p:05Triage>03High a:03demon [00:17:04] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856046 (10Smalyshev) [00:17:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2856047 (10Pchelolo) Hm, so this is a bit of a chicken and egg problem. I can't set up te... [00:19:29] 10Gerrit: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2856061 (10demon) I'm *thinking* this has something to do with the external indexing. I could be wrong though. [00:35:29] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856072 (10demon) 05Open>03Resolved This *shouldn't* happen anymore, but lemme know if it does. [00:36:41] 10Gerrit: Gerrit produces 503 on some patches - https://phabricator.wikimedia.org/T152648#2856023 (10Pchelolo) Had an issue with that deploying #restbase but it seems to work fine now. Kudos! [00:36:45] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856076 (10demon) So, couple of other users have also reported (@awight @kaldari). I think it's a capitalization bug that **should** have been fixed already but does not appear to have been. Trying different capital... [00:40:19] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [00:41:58] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856084 (10Eloquence) I get the same error message regardless of capitalization. The number (4136 in my original report) ticks up with each attempt. [00:43:18] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856098 (10demon) >>! In T152640#2856084, @Eloquence wrote: > I get the same error message regardless of capitalization. The number (4136 in my original report) ticks up with each attempt. Yeah, the number increasi... [00:43:56] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856099 (10Eloquence) I don't know whether this was the case before, but both 'erik' and 'eloquence' users seem to exist, producing the same results: https://gerrit.wikimedia.org/r/#/q/owner:erik+status:merged http... [00:44:34] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856100 (10Eloquence) However, when I attempt to login as 'erik' I get a different error message: 'Invalid username or password.' [01:23:46] Hi! Help ssh'ing into beta cluster? getting Permission denied (publickey) from ssh deployment-tin.eqiad.wmflabs [01:23:50] thx in advance!!! [01:26:30] (I can never remember if this is the right place to ask about betacluster stuff...) [01:30:34] does that happen when you ssh to other labs hosts AndyRussG? [01:31:18] Krenair: yep! [01:31:22] For example, deployment-fluorine02.deployment-prep.eqiad.wmflabs [01:31:29] I'm a member of the project... [01:31:47] deployment-tin.deployment-prep.eqiad.wmflabs ? [01:35:44] Krenair: yep same... [01:36:21] Same key that I use for gerrit [01:36:26] (which works) [01:36:57] AndyRussG, try it now [01:37:38] 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856211 (10Paladox) https://groups.google.com/forum/#!msg/repo-discuss/U6qM_4j5wG0/TEZi-1RkCQAJ [01:37:41] Krenair: no same ;( [01:38:12] AndyRussG, alright, please dump a copy of your ssh config as well as everything shown when you ssh -vvv to the host [01:38:39] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856212 (10demon) Filed upstream: https://bugs.chromium.org/p/gerrit/issues/detail?id=5090 [01:42:13] Krenair: https://tools.wmflabs.org/paste/view/12671c93 [01:43:34] Maybe it's this part that's forcing the wrong identify file? [01:43:36] https://tools.wmflabs.org/paste/view/12671c93 [01:43:39] Ah sorry [01:43:46] Host *.wikimedia.org *.wmnet !gerrit.wikimedia.org !tellurium.wikimedia.org [01:43:47] User andyrussg [01:43:49] ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org [01:43:51] IdentitiesOnly yes [01:43:53] IdentityFile ~/.ssh/wmf_cluster [01:44:08] Ah no that shouldn't be a problem [01:44:15] AndyRussG, line 32... [01:44:23] Since this is .wmflabs [01:44:33]

@primary.bastion.wmflabs.org? [01:44:43] Wooooops heheheheh [01:45:22] it might not be causing issues here [01:46:33] Krenair: yeah no change... I think it's a bit of lost copypasta from just a few minutes when I was trying to fix the problem, so the problem was from before that [01:46:50] if you run `ssh-add -L`, does it show 'ssh-dss AAAAB3NzaC1kc3MAAACBALpmfMqPIAVIjy9DZgN4ltwB6zK0kzKj5lTBnSHG/uGDgSW6GMEmFgXRlyt4hQbrXGhi3PjxbWbbF26k1Bv1bGJ8pvWsOZ8OBO8KEFtv86/NSs2V6W2xW5qEQ8h875uuR/rLYljZ2bVziTZAFF5rQszg0KUE8pROODbkXPCiNLJlAAAAFQDhSpV4J1ZwkD8u7OZYuCMLwlpz3wAAAIAyYLsbJsZUPy1rLgtQiDqwWOqbn/RZIXbxTXXhvKXrHxH60JGvMt2gXJ4+AqAVnsZvKRkzW0G/0XoRS9JsK5kfrJdgtZHp47ARyjCc0KKo1UWIUB3RkhYwP0o9zX3Qy3febiyZRcw4zNLeFbERakKnTZvomrrVMndXfqlMaMVXJ [01:46:50] AAAAIB2vfkR0MCMgeQ1KCO7FxExkT0Dq+fgzVVLyKETMOYGzT8GITTTV8mVQERwlfGnMkb94+kDottGX1XTAso8tb27ocb3hKhGPdyXhP8EYBY8MVLxzx5XiDDS0GMJY2+Cuacwy/8duFgu37EbMHZVScWm/Z/EvirFmWlYOCg0sUE9LA== debian1@debian3' ? [01:52:21] AndyRussG? [01:53:51] Krenair: sorry... minor interruptions plaguing me here :/ It says that key [01:53:57] but with a different e-mail at the end [01:54:23] the comment shouldn't matter? [01:54:28] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856225 (10demon) Adding 2 other affected users per error_log grepping: @pppery @Papaul So far it's looking like there might be a manual workaround, but i... [01:54:31] Ah wait mebbe I'm wrong [01:54:33] I think it doesn't matter [01:55:13] AndyRussG, You can ssh to primary.bastion.wmflabs.org though right? [01:55:53] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856249 (10Paladox) p:05High>03Unbreak! Changing to unbreak as this could affect a lot of users, not all but could be more then 5+ [01:56:04] 06Release-Engineering-Team, 03Scap3 (Scap3-MediaWiki-MVP): /srv/mediawiki on tin not being updated when using scap sync-file - https://phabricator.wikimedia.org/T152005#2856252 (10demon) [01:56:07] 03Scap3 (Scap3-MediaWiki-MVP): ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2856254 (10demon) [01:56:51] Krenair: nope... Also, I goofed that key isn't shown when I say ssh-add -L [01:57:08] 10Beta-Cluster-Infrastructure, 03Scap3 (Scap3-MediaWiki-MVP), 06Operations, 07WorkType-NewFunctionality: etcd/confd is not started on beta cluster Varnish caches - https://phabricator.wikimedia.org/T116224#2856262 (10demon) [01:58:21] AndyRussG, okay, from reading your ssh -vvv log and from that, this issue is way before you hit deployment-prep [01:58:24] you can't even get into labs [01:58:37] o/ hey all. I pushed a patch into Grrrit earlier today and it has since disappeared. I have a local copy so no worries but do you want to check it out before I re-push? The link is in the comments here: https://gerrit.wikimedia.org/r/#/c/325695/3/app/src/main/java/org/wikipedia/descriptions/DescriptionEditTutorialViewSecond.java [01:58:38] so, we move to -labs [01:59:32] niedzielski: I'm pretty sure it's there, I think it's an indexing issue. [01:59:53] ostriches: ah, so if i wait it'll probably come around? [02:00:06] I'm hoping, I wouldn't worry just yet about it [02:00:10] It's in the database, for sure. [02:00:29] ostriches: ok, sounds good. i have a local copy so no worries. thanks! [02:00:36] Being tracked at https://phabricator.wikimedia.org/T152642 [02:01:28] 👍 [02:04:10] Krenair: yeah! huh I changed Gerrit key like 6 months ago, I was assuming that it's somehow coordinated with labs [02:04:17] -labs [02:04:20] But https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack shows the old key, the one you pasted [02:04:31] I should just update it there, no? [02:04:38] not discussing it here, -labs [02:04:50] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856274 (10demon) p:05Unbreak!>03High It could also affect only 5. And we can't exactly fix it without either upstream issuing a fix or every one of th... [02:04:52] Krenair: ah sorry I understand [02:16:31] 10Gerrit, 07Regression: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2856287 (10demon) p:05Triage>03High a:03demon [02:16:55] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T150465#2856290 (10demon) 05Open>03Resolved [02:26:27] 10Beta-Cluster-Infrastructure, 03Scap3 (Scap3-MediaWiki-MVP), 06Operations, 07WorkType-NewFunctionality: etcd/confd is not started on beta cluster Varnish caches - https://phabricator.wikimedia.org/T116224#2856292 (10Krenair) 05Open>03Resolved a:03Krenair https://wikitech.wikimedia.org/w/index.php?ti... [02:35:28] meh [02:35:36] I should probably go through beta-cluster-infra again at some stage [02:35:42] likely more low-hanging fruit lying about [02:46:15] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856311 (10demon) Adding @matthewrbowker per logging. [03:00:14] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856352 (10demon) Looks like 11 users total, the following 10 + Awight (he was mid-live-hack-debugging): ``` gerrit> select account_id, full_name from acc... [03:06:36] (03PS1) 10Legoktm: Add experimental 'mediawiki-core-php70-phan' job [integration/config] - 10https://gerrit.wikimedia.org/r/325873 (https://phabricator.wikimedia.org/T132636) [03:06:50] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856356 (10demon) [03:07:19] (03CR) 10jenkins-bot: [V: 04-1 C: 040] Add experimental 'mediawiki-core-php70-phan' job [integration/config] - 10https://gerrit.wikimedia.org/r/325873 (https://phabricator.wikimedia.org/T132636) (owner: 10Legoktm) [03:08:57] 03:07:00 Could not fetch URL https://pypi.python.org/simple/nose/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) - skipping [03:09:08] hmm [03:14:26] 10Continuous-Integration-Infrastructure: CI machines think pypi.python SSL certificate is expired - https://phabricator.wikimedia.org/T152660#2856358 (10Legoktm) [03:16:22] legoktm: Outdated ca-certificates? [03:16:41] ostriches: according to sudo apt list --upgradable there's no update available [03:17:01] legoktm@integration-slave-jessie-1002:~$ dpkg -l | grep ca-cert [03:17:01] ii ca-certificates 20141019+deb8u1 all Common CA certificates [03:17:14] 20141019+deb8u1 [03:17:18] 2014? [03:18:00] Eh, that's current. [03:20:13] yeah I'm not sure... [03:20:22] wait [03:20:24] now it works??? [03:20:32] legoktm@integration-slave-jessie-1002:~$ curl https://pypi.python.org/simple/nose/ [03:20:32] Links for nose

Links for nose

nose-1.3.4-py3-none-any.whl
[03:20:32] etc [03:21:50] whaaaaaat https://integration.wikimedia.org/ci/job/integration-config-tox-jessie/2314/console it's working now -.- [03:22:09] ostriches: I swear it was broken [03:23:44] 10Continuous-Integration-Infrastructure: CI machines think pypi.python SSL certificate is expired - https://phabricator.wikimedia.org/T152660#2856375 (10Legoktm) 05Open>03Resolved a:03Legoktm It appears to be working now...? But it was definitely broken based on my curl command was well as the first two fa... [03:24:09] (03CR) 10Legoktm: [C: 032] Add experimental 'mediawiki-core-php70-phan' job [integration/config] - 10https://gerrit.wikimedia.org/r/325873 (https://phabricator.wikimedia.org/T132636) (owner: 10Legoktm) [03:25:15] RoanKattouw, ^ it works now [03:25:36] (03Merged) 10jenkins-bot: Add experimental 'mediawiki-core-php70-phan' job [integration/config] - 10https://gerrit.wikimedia.org/r/325873 (https://phabricator.wikimedia.org/T132636) (owner: 10Legoktm) [03:26:32] !log deploying https://gerrit.wikimedia.org/r/325873 [03:26:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:29:34] Yay thanks Krenair [04:04:27] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10santhosh) Me too: {F5008906} [04:07:46] (03PS1) 10Legoktm: phan: Switch to dev-master, 0.6.0 is too old [integration/config] - 10https://gerrit.wikimedia.org/r/325876 [04:09:54] (03CR) 10Legoktm: [C: 032] "already deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/325876 (owner: 10Legoktm) [04:10:40] (03PS2) 10Legoktm: phan: Switch to dev-master, 0.6.0 is too old [integration/config] - 10https://gerrit.wikimedia.org/r/325876 [04:10:57] (03CR) 10Legoktm: [C: 032] phan: Switch to dev-master, 0.6.0 is too old [integration/config] - 10https://gerrit.wikimedia.org/r/325876 (owner: 10Legoktm) [04:14:01] (03Merged) 10jenkins-bot: phan: Switch to dev-master, 0.6.0 is too old [integration/config] - 10https://gerrit.wikimedia.org/r/325876 (owner: 10Legoktm) [04:17:56] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #227: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/227/ [05:26:20] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/325877/ onto integration-puppetmaster01 [05:26:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:31:10] !log legoktm@integration-saltmaster:~$ sudo salt '*jessie*' cmd.run 'puppet agent -tv' [05:31:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:41:08] legoktm: do you know how the magic mediawiki/exensions repo gets updated? [05:41:09] 10Beta-Cluster-Infrastructure: Beta-cluster not updating extensions - https://phabricator.wikimedia.org/T152663#2856457 (10Yurik) [05:41:15] it's gerrit magic [05:41:27] it looks like it stopped getting bumps with the upgrade :/ [05:41:31] :( [05:41:46] filed ^^ [05:41:50] last change 11 ours ago -- https://github.com/wikimedia/mediawiki-extensions/commits/master [05:42:16] 10Gerrit: Gerrit not updating submodules in mediawiki/extensions - https://phabricator.wikimedia.org/T152663#2856469 (10Legoktm) [05:43:19] yurik: once the magic repo is fixed beta will get fixed [05:43:47] but I bet ostriches is out drinking and trying to forget that he upgraded gerrit today [05:43:49] * yurik grabs a woodoo mask and a tambourine ... [05:44:23] he should have known better than to touch it [05:49:02] https://www.gerritcodereview.com/releases/2.13.md#Access-control-for-git-submodule-subscriptions [05:49:49] "To prevent potential security breaches as described in issue 3311, it is now only possible for a project to subscribe to a submodule if the submodule explicitly allows itself to be subscribed." [05:50:52] hmm [05:50:57] "Note that when upgrading from an earlier version of Gerrit, permissions for any existing subscriptions will be automatically added during the database schema migration." did that part not work? [05:51:01] [allowSuperproject ""] [05:51:01] matching = [05:51:12] I doubt all submodules got those [05:52:11] Clearly not. [05:52:12] can we set that in the parent mediawiki/extensions project so all extensions under it inherit? [05:52:23] No idea. [05:52:44] The configuration is inherited from parent projects, such that you can have a configuration in the "All-Projects" project like: [05:53:22] Ok, so we should be able to fix that at All-Projects, then re-subscribe them? [05:54:11] let's try it? [05:54:13] :) [05:54:38] [allowSuperproject "my-only-superproject"] [05:54:38] matching = refs/heads/*:refs/heads/* [05:54:43] I wonder if I can do "*" [05:56:01] At the very least, we could set it at mediawiki/extensions for now. [05:57:06] mediawiki/skins too, but I think less people use that [05:57:09] [allowSuperproject ""] [05:57:09] all = refs/heads/*:refs/heads/* [05:59:23] Needs skins, extensions & vendor [05:59:28] For wmf/ branch support [06:00:27] https://gerrit.wikimedia.org/r/#/c/325884/ [06:02:19] https://gerrit.wikimedia.org/r/#/c/325885/ [06:02:29] legoktm: beta does ;) [06:03:12] ostriches: or did you want +2's? [06:03:53] Let's go ahead, worst case it doesn't help :) [06:05:11] Now, if they start subscribing again automatically or if mw/extensions and mw/skins need some manual pushing again... [06:05:11] do we need a dummy patch to extensions/examples or something now? [06:06:01] I +2'd https://gerrit.wikimedia.org/r/#/c/255502/ [06:06:02] heeh [06:06:12] lol failed tests [06:06:38] Aww conflict [06:06:51] [22:06:29] (PS1) Legoktm: Add comment [extensions/examples] - https://gerrit.wikimedia.org/r/325886 [06:26:48] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #201: 04FAILURE in 1 hr 46 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/201/ [06:28:36] 10Gerrit: Gerrit not updating submodules in mediawiki/extensions - https://phabricator.wikimedia.org/T152663#2856457 (10Legoktm) Debugged with @demon on IRC: Caused by: Updated config: https://gerrit.wikimedia.or... [08:32:09] ok [08:32:14] going to upgrade the Jessie image [08:32:24] which would be faster to boot due to some DHCP oddity [08:32:26] \O/ [08:32:59] the one line fix being : sudo rm -fv /var/lib/dhcp/dhclient.*.leases [08:33:00] !! [08:33:42] !sal [08:33:42] https://tools.wmflabs.org/sal/releng [08:39:06] morning hashar [08:39:25] erik and I have mostly gotten phan running against mw-core https://gerrit.wikimedia.org/r/#/c/296860/ :) [08:48:56] !log Image ci-jessie-wikimedia-1481186016 in wmflabs-eqiad is ready T113342 [08:49:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:49:02] legoktm: awesome!!! [08:49:10] legoktm: thanks to sury.org and php7 isn't it ? [08:49:36] yep [08:49:53] it's not running on nodepool yet because I had to cherry-pick a puppet patch for more packages [08:50:20] a workaround is to use the puppet manifest in integration/config and refresh the snapshot [08:50:34] but that is ugly :} [08:51:38] legoktm: seems like all your puppet.git patches are merged in [08:51:53] the puppet patch is https://gerrit.wikimedia.org/r/325877 by erik [08:51:59] ohhh [08:52:29] yeah that is the same set of packages we have for mediawiki::packages::php5 (or some class like that) [08:56:14] legoktm: we have been experimenting with Docker. [08:56:31] legoktm: so maybe next year we will switch to developers defining their packages/build env in their repo Dockerfile [08:56:36] that will solve a lot of madness [08:56:36] I saw the ticket got updated, but haven't looked to see how it works yet :D [08:56:44] Dan is leading it [08:57:07] he has set a very nice proof of concept with Mukunda for the scap repo and malu (javascript helper for selenium) [08:58:15] gehel: hey there! do you get some spare time to land a puppet patch that add a few packages please ? :} [08:58:40] https://gerrit.wikimedia.org/r/#/c/325877/ is to add php7 extensions on CI. Will let us run some code analytic utility for mediawiki/core CirrusSearch etc [08:59:43] hashar: I'm trying to get a new screen for my laptop. Should be back home and available in ~1h... [08:59:57] gehel: no worries! good luck with the screen hunt! [09:20:47] Trying to find someone to deploy the Account creation throttling exception in https://gerrit.wikimedia.org/r/325899 for https://phabricator.wikimedia.org/T152669 [09:21:18] (Urbanecm also asked in #operations) [09:21:37] (ah, lego is checking. alright.) [09:41:43] (03PS1) 10Legoktm: Move mediawiki-core-php70-phan into test/gate-and-submit as nonvoting [integration/config] - 10https://gerrit.wikimedia.org/r/325903 [09:43:53] (03CR) 10Legoktm: [C: 032] Move mediawiki-core-php70-phan into test/gate-and-submit as nonvoting [integration/config] - 10https://gerrit.wikimedia.org/r/325903 (owner: 10Legoktm) [09:44:40] (03Merged) 10jenkins-bot: Move mediawiki-core-php70-phan into test/gate-and-submit as nonvoting [integration/config] - 10https://gerrit.wikimedia.org/r/325903 (owner: 10Legoktm) [09:45:18] !log deploying https://gerrit.wikimedia.org/r/325903 [09:45:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:48:01] (03PS1) 10Legoktm: phan: Add the standard composer flags to the update command [integration/config] - 10https://gerrit.wikimedia.org/r/325904 [09:48:58] (03CR) 10Legoktm: [C: 032] phan: Add the standard composer flags to the update command [integration/config] - 10https://gerrit.wikimedia.org/r/325904 (owner: 10Legoktm) [09:50:37] (03Merged) 10jenkins-bot: phan: Add the standard composer flags to the update command [integration/config] - 10https://gerrit.wikimedia.org/r/325904 (owner: 10Legoktm) [09:50:48] legoktm: I usually avoid adding non voting jobs to gate-and-submit :D [09:50:59] not a big deal though since I guess it will be voting soonish [09:51:57] ah okay [09:52:01] yeah, should be a day or two I think [09:52:06] I'm writing up a comment about next steps [09:52:09] the idea is that it consumes an instance / an executor slot for nothing [09:52:15] folks having the report via the test pipeline already [09:52:19] but really it is not a big deal [09:52:48] the key point is that phan integration is almost done :} [09:53:42] 00:00:45.556 + ./tests/phan/bin/phan [09:53:42] 00:02:51.129 Finished: SUCCESS [09:53:43] \O/ [09:58:44] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Install php7 and the php-ast extension so etsy/phan can be run from jenkins - https://phabricator.wikimedia.org/T132636#2856743 (10Legoktm) In a few minutes, the phan job will be set up non-voting against all mw/core patches. Here are the next steps... [09:59:02] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2856744 (10Legoktm) [10:00:15] hashar: I'm back... what do you need? [10:00:25] got a new screen ? :} [10:00:42] nope, the laptop is sent back for repair, but I have a desktop @home [10:00:54] E Bernarhdson / Kunal have been working on adding a new PHP static analysis code written by esty [10:01:00] it requires PHP7 to run [10:01:08] hashar: yep, cool stuff! [10:01:19] analyses mediawiki code which in turns requires a bunch of Zend extensions to be added (eg memcached) [10:01:23] gehel: we need a few more packages to be installed :) https://gerrit.wikimedia.org/r/#/c/325877/ [10:01:26] so https://gerrit.wikimedia.org/r/#/c/325877/ adds the php7 packages to the CI machine [10:01:42] ok, let me have a look... [10:01:49] contint::packages::php being only on labs instances :D [10:02:43] we could have added a mediawiki::packages::php class that would get a list of versions (5 and/or 7) [10:02:56] but I dont think we will move prod to php 7 at anytime :D [10:03:10] so there is a slight code duplication, but that is not the end of the world [10:03:40] hashar: looks good enough and simple enough to me [10:03:48] want me to merge this right now? [10:03:55] if at all possible yes :} [10:04:16] coming up! [10:06:53] thanks :)) [10:08:09] legoktm, hashar: merged... let me know if there is any issue [10:08:17] awesome [10:08:34] I am in a call with zeljko, refreshing the Jessie snapshot [10:10:42] refreshing jessie image [10:18:08] done [10:18:17] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856754 (10Aklapper) Please try replacing your capital letters by small letters and report back. I ran into the same problem when using "Aklapper", there w... [10:18:59] legoktm: the Jessie snapshot should have the php7 packages.I am purging the old instances manually [10:21:25] !log Refreshing Nodepool base image for Trusty. Was blocked on a mariadb upgrade, should also acquire network faster T113342 [10:21:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:28:22] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2856768 (10santhosh) > Please try replacing your capital letters by small letters and report back. Same error with small letters too. [10:31:48] !log Image ci-trusty-wikimedia-1481192772 in wmflabs-eqiad is ready [10:31:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:38:37] !sal [10:38:37] https://tools.wmflabs.org/sal/releng [10:44:33] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2856785 (10hashar) For Jessie: ``` $ grep 445011 /var/log/nodepool/debug.log 08:59:06,747 DEBUG node... [10:46:46] (03PS1) 10Legoktm: Move mediawiki-core-php70-phan to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/325912 [10:48:00] fast booooot https://phabricator.wikimedia.org/P4591 [10:48:19] Dec 08 10:46:45 hashar-jessie systemd[1]: Startup finished in 7.102s (kernel) + 11.916s (userspace) = 19.018s. [10:49:00] https://phabricator.wikimedia.org/P4591 [10:49:08] the dhcp being "instant" now [10:49:09] \O/ [11:14:40] (03PS1) 10Legoktm: Initial debianization [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325914 (https://phabricator.wikimedia.org/T152577) [11:15:01] (03Abandoned) 10Legoktm: Initial debianization [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325914 (https://phabricator.wikimedia.org/T152577) (owner: 10Legoktm) [11:15:21] (03PS2) 10Zfilipin: WIP RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) [11:16:04] (03PS2) 10Legoktm: Initial debianization [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/325726 (https://phabricator.wikimedia.org/T152577) [11:23:34] (03PS3) 10Zfilipin: WIP RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) [11:31:42] 03Scap3 (Scap3-MediaWiki-MVP), 10Electron-PDFs, 03WMDE-QWERTY-Team-Board: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2856833 (10Tobi_WMDE_SW) [11:32:09] PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:05:57] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Addshore) me too! {F5010492} [12:06:08] (03PS4) 10Zfilipin: WIP RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) [12:07:08] RECOVERY - Puppet run on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [12:13:59] (03CR) 10Hashar: [C: 031] "Lets deploy that and iterate! :}" [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) (owner: 10Zfilipin) [12:14:30] (03PS5) 10Zfilipin: RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) [12:16:00] good news the Jessie instances seems to be READY in just 50 seconds :D [12:16:00] (03PS6) 10Zfilipin: RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) [12:16:10] (03CR) 10Zfilipin: [C: 032] RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) (owner: 10Zfilipin) [12:17:50] (03Merged) 10jenkins-bot: RSpec tests for Echo Mention notification [integration/config] - 10https://gerrit.wikimedia.org/r/324687 (https://phabricator.wikimedia.org/T146916) (owner: 10Zfilipin) [12:20:00] we should probably set that BUNDLE_PATH in the jenkins profile [12:25:42] addshore: are you around? :D [12:25:48] yup [12:25:57] addshore: for the Gerrit login authentication issue would you mind copy pasting the output of https://gerrit.wikimedia.org/r/#/settings/ ? [12:26:08] wondering what the username and Full Name are [12:26:20] Account ID 106 [12:26:26] pretty sure it is bad upper case / lower case normalization [12:26:27] Username [12:26:27] addshore [12:26:34] Full Name Addshore [12:26:55] awesome thanks [12:46:42] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10hashar) Taking addshore as an example: https://gerrit.wikimedia.org/r/#/settings/ and LDAP overview: | Ldap | Gerrit conf | Gerrit settings pa... [12:48:10] [= [13:25:57] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Niharika) Hi, I cannot login to Gerrit either. :( My username is niharika29. [13:27:45] 10Continuous-Integration-Infrastructure: CI machines think pypi.python SSL certificate is expired - https://phabricator.wikimedia.org/T152660#2856358 (10hashar) That was an incident on pypi side //Certificate Expired on PyPI// https://status.python.org/incidents/3fvl2mdpj8f4 . Their CDN Fastly had some intermitt... [13:59:14] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2857022 (10hashar) 05Open>03Resolved a:03hashar Both Jessie and Trusty have caches populated now: ``` $ find... [14:03:01] (03PS1) 10Hashar: [integration/uprightdiff] non voting debian-glue job [integration/config] - 10https://gerrit.wikimedia.org/r/325930 (https://phabricator.wikimedia.org/T152577) [14:23:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2857046 (10Ottomata) FYI, I’ve made some slight modifications to Petr’s kafka .sh script... [14:28:42] 10Browser-Tests-Infrastructure, 06Maps (Kartographer), 15User-zeljkofilipin: Set up Selenium tests for Kartographer extension - https://phabricator.wikimedia.org/T152683#2857050 (10zeljkofilipin) [14:29:07] 10Browser-Tests-Infrastructure, 07JavaScript, 06Maps (Kartographer), 15User-zeljkofilipin: Set up Selenium tests for Kartographer extension - https://phabricator.wikimedia.org/T152683#2857063 (10zeljkofilipin) p:05Triage>03Normal [14:34:41] (03CR) 10Hashar: "Looks great! A few things inline, most important is: should we just put the /debian/ directory in master branch since that is a Debian n" (037 comments) [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/325726 (https://phabricator.wikimedia.org/T152577) (owner: 10Legoktm) [14:41:53] zeljkof: for echo/rspec, have you tried bumping mediawiki-selenium ? [14:42:05] looks like something is off and the code doesn't find the environments.yml file or it is missing [14:42:47] hashar: sorry, got distracted with other things, will do, that should be an easy fix [14:45:32] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857076 (10hashar) [14:47:31] hashar: done [14:48:07] (03PS1) 10Hashar: Jobs for mediawiki/services/change-propagation [integration/config] - 10https://gerrit.wikimedia.org/r/325933 (https://phabricator.wikimedia.org/T152684) [14:48:37] zeljkof: and there might be something screwed up / related to environments.yml [14:49:59] 10Browser-Tests-Infrastructure, 07JavaScript, 06Maps (Kartographer), 15User-zeljkofilipin: Set up Selenium tests for Kartographer extension - https://phabricator.wikimedia.org/T152683#2857114 (10zeljkofilipin) [[ https://gerrit.wikimedia.org/r/#/c/323401/ | 323401 ]] is an example for mediawiki/core. [14:51:37] (03CR) 10Hashar: [C: 032] Jobs for mediawiki/services/change-propagation [integration/config] - 10https://gerrit.wikimedia.org/r/325933 (https://phabricator.wikimedia.org/T152684) (owner: 10Hashar) [14:54:01] (03PS2) 10Hashar: Jobs for mediawiki/services/change-propagation [integration/config] - 10https://gerrit.wikimedia.org/r/325933 (https://phabricator.wikimedia.org/T152684) [14:55:59] zeljkof: https://integration.wikimedia.org/ci/job/mwext-mw-rspec-jessie/3/console [14:56:04] still seems to use 1.7.2 :( [14:56:35] I think I know what is the problem, just a sec [14:56:48] (03CR) 10Hashar: [C: 032] Jobs for mediawiki/services/change-propagation [integration/config] - 10https://gerrit.wikimedia.org/r/325933 (https://phabricator.wikimedia.org/T152684) (owner: 10Hashar) [14:58:18] my mistake, forgot to push Gemfile.lock [14:58:28] ok, meetings for the next two hours or so [14:58:31] (03Merged) 10jenkins-bot: Jobs for mediawiki/services/change-propagation [integration/config] - 10https://gerrit.wikimedia.org/r/325933 (https://phabricator.wikimedia.org/T152684) (owner: 10Hashar) [15:02:25] oh [15:02:40] just thought again how to cache the package manager binary compiles [15:02:46] and maybe we could use ccache :} [15:05:33] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.21.29) [15:10:56] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Use ccache for CI compilation? - https://phabricator.wikimedia.org/T152688#2857163 (10hashar) [15:12:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857175 (10hashar) Both builds fail with: ``` > sh test/utils/clean_kafka.sh t... [15:13:53] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2857176 (10Paladox) @Niharika hi could you try Niharika29 please? Note that it is a capital n not lower case. [15:21:09] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857179 (10hashar) Trusty images have it because `netcat-openbsd` is part of th... [15:25:45] (03PS1) 10Hashar: dib: provision netcat-openbsd [integration/config] - 10https://gerrit.wikimedia.org/r/325937 (https://phabricator.wikimedia.org/T151469) [15:26:37] (03CR) 10Hashar: [C: 032] dib: provision netcat-openbsd [integration/config] - 10https://gerrit.wikimedia.org/r/325937 (https://phabricator.wikimedia.org/T151469) (owner: 10Hashar) [15:27:04] spaghetti sauce! [15:27:27] (03Merged) 10jenkins-bot: dib: provision netcat-openbsd [integration/config] - 10https://gerrit.wikimedia.org/r/325937 (https://phabricator.wikimedia.org/T151469) (owner: 10Hashar) [15:28:55] !log Updating Nodepool Jessie image to ship `netcat` T151469 T152684 [15:29:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:32:42] Notice: /Stage[main]/Main/Package[netcat-openbsd]/ensure: ensure changed 'purged' to 'present' [15:33:48] !log Image ci-jessie-wikimedia-1481210905 in wmflabs-eqiad is ready : Notice: /Stage[main]/Main/Package[netcat-openbsd]/ensure: ensure changed 'purged' to 'present' [15:33:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:39:36] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Huji) Add me to the list as the 12th person (account 4254, username = Huji). [15:44:22] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2857201 (10Huji) The combination of "name already in use" and the fact that account #1 is incremented by 1 each time makes me think: is Gerrit trying to re... [16:09:13] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2857225 (10Paladox) Yep @Huji your right, but this was supposedly fixed in gerrit 2.13.2 but seems it wasen't. [16:24:34] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Reedy) ``` Cannot assign user name "reedy" to account 4256; name already in use. ``` Logging in as reedy rather than Reedy WFM though to fix it [16:28:32] (03PS1) 10Hashar: Move change-propagation jobs to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/325952 (https://phabricator.wikimedia.org/T152684) [16:31:10] (03CR) 10Hashar: [C: 032] Move change-propagation jobs to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/325952 (https://phabricator.wikimedia.org/T152684) (owner: 10Hashar) [16:33:20] (03Merged) 10jenkins-bot: Move change-propagation jobs to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/325952 (https://phabricator.wikimedia.org/T152684) (owner: 10Hashar) [16:36:57] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857284 (10hashar) I gave it a try on https://gerrit.wikimedia.org/r/#/c/325934... [16:38:53] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2857286 (10hashar) @Pchelolo since change-propagation already has the Kafka related glue,... [17:25:19] 10Gerrit, 13Patch-For-Review: Gerrit not updating submodules in mediawiki/extensions - https://phabricator.wikimedia.org/T152663#2857381 (10Paladox) I think this is a bug. I have been trying all day on two test sites and they are not working. running gerrit 2.13 too. [17:31:28] PROBLEM - Host deployment-trending is DOWN: CRITICAL - Host Unreachable (10.68.19.98) [17:43:30] gerrit issues? "Cannot assign user name "gwicke" to account 4272; name already in use." [17:43:47] this happens on login [17:43:54] gwicke hi known problem [17:43:59] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2857441 (10EBernhardson) Puppet match merged. Moving the version out of CI into mw/core might be difficult. Currently with composer if we were to add phan as a... [17:43:59] every login attempt increments the number by one [17:44:09] gwicke see https://phabricator.wikimedia.org/T152640 [17:44:10] please [17:44:19] kk, thx! [17:44:26] gwicke: Try "Gwicke" [17:44:32] (dumb, yes, I know, but is working for some people) [17:44:45] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2857443 (10EBernhardson) Quick response. morria over at phan cut an 0.7.0 release today. https://github.com/etsy/phan/issues/436 [17:45:12] ostriches: thanks for the hint, "GWicke" worked [17:45:34] Yeah, it's super sensitive to capitalization now. [17:46:02] That's working for *some* people [17:46:03] :) [17:46:38] I might be able to work around this entirely by swapping to case-insensitive logins, but I'm not sure all the fallout. The docs are a little vague on this (surprise) [17:48:09] :/ [17:53:10] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2857481 (10Legoktm) > Moving the version out of CI into mw/core might be difficult. Currently with composer if we were to add phan as a dev requirement composer... [17:57:35] (03PS1) 10BryanDavis: Add tox tests for labs/tools/stashbot [integration/config] - 10https://gerrit.wikimedia.org/r/325974 [18:05:36] marxarelli: hey I wanted to say I really wasn't intending to dump on your current experiment at all, I hope it didn't come across that way [18:05:58] chasemp: not at all :) [18:06:13] is your shell name in labs marxarelli? [18:06:20] dduvall [18:06:30] tricky [18:08:45] marxarelli: do you want to meet up early next week we can compare notes then? [18:10:41] chasemp: next week sounds good to me [18:12:38] chasemp: i'll schedule something [18:13:07] monday is pretty crap for me, but first thing tuesday? [18:13:21] was just looking at that :) so many meetings [18:20:24] chasemp: invitation sent. i can also do 9am pst but i wasn't sure if you would be attending the tech managers meeting [18:23:49] usually yes, this works [18:28:21] marxarelli: for your reviewing pleasure -- https://gerrit.wikimedia.org/r/#/c/320277/ -- I'm inclined to merge it on the basis of fixing 1.9.0 breakage alone, but have only tested myself on OSX and in Tool Labs. [18:28:45] * bd808 still doesn't have a windows test rig for mw-vagrant [18:29:02] bd808: shit. i just nuked mine to save disk space :) [18:29:21] i'll review your patch today for sure [18:29:29] meant to get to it earlier in the week [18:29:56] one question about it though: should we just get rid of setup altogether? [18:30:40] basically s/setup.sh/vagrant config --required/g the README [18:31:27] we could, yeah [18:32:36]