[00:23:21] (03PS4) 10Krinkle: GeoCrumbs no longer depends on CustomData in master [integration/config] - 10https://gerrit.wikimedia.org/r/300068 (owner: 10Jforrester) [00:23:25] (03CR) 10Krinkle: [C: 032] GeoCrumbs no longer depends on CustomData in master [integration/config] - 10https://gerrit.wikimedia.org/r/300068 (owner: 10Jforrester) [00:24:25] (03Merged) 10jenkins-bot: GeoCrumbs no longer depends on CustomData in master [integration/config] - 10https://gerrit.wikimedia.org/r/300068 (owner: 10Jforrester) [00:29:00] 06Release-Engineering-Team, 10Phabricator: Requesting access for @Danny_B to Phame - https://phabricator.wikimedia.org/T142142#2525357 (10Danny_B) >>! In T142142#2524571, @greg wrote: > For all access requests there needs to be a "why" included :) Sorry... That's the disadvantage of knowing I've already mentio... [00:31:33] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/300068 [00:31:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:38:25] (03Draft2) 10Jforrester: [ParserMigration] Run composer test [integration/config] - 10https://gerrit.wikimedia.org/r/303113 [00:39:28] (03CR) 10Krinkle: [C: 032] [ParserMigration] Run composer test [integration/config] - 10https://gerrit.wikimedia.org/r/303113 (owner: 10Jforrester) [00:39:40] James_F: :) [00:40:33] (03Merged) 10jenkins-bot: [ParserMigration] Run composer test [integration/config] - 10https://gerrit.wikimedia.org/r/303113 (owner: 10Jforrester) [00:41:28] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/303113 [00:41:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:41:33] Thanks Krinkle. [00:42:32] James_F: Why npm? [00:42:58] Krinkle: Why not. It's a no-op. [00:43:05] Not really [00:43:14] Lego was in favour. [00:43:51] There is no .js files or package.json [00:44:22] So it fails [00:44:48] I thought we fixed that so it skips when there's no package.json? [00:45:15] that's the first I hear [00:45:54] We certainly did for composer, right? [00:45:56] Nope, you always need package.json for npm [00:46:00] it is for composer yes [00:46:01] Meh. [00:46:07] Sorry. [00:46:13] I assumed we'd done that. [00:46:33] It may be possible but im not sure [00:47:08] it's possible but it's also a waste of nodepool resources [00:47:17] Yep [00:47:20] They're not separate steps like on Travis [00:47:35] Oh [00:47:45] I'll remove for now. [00:48:04] (03PS1) 10Krinkle: Remove unused 'npm' job for ParserMigration [integration/config] - 10https://gerrit.wikimedia.org/r/303115 [00:48:12] James_F: +1 ? [00:48:22] Actually does that repo have json files [00:48:26] Krinkle ^^ [00:48:50] If so i think we might as well add package.json to the repo and test the json files, thats what were doing with the other files [00:49:22] + i have been going through the repos doing that, but i have stop for a few months since i have been doing other things [00:49:34] It has extension.json if nothing else. [00:49:50] Yep we can use jsonlint for that [00:50:02] Kk. One second. [00:50:08] Thanks [00:50:24] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2525383 (10Krinkle) [00:50:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2525395 (10Krinkle) p:05Triage>03Unbreak! [00:51:15] We can also lint those with a composer test. [00:51:28] Or we can even run 'npm install && npm test' from one of the composer-scripts-test array items [00:51:40] That's what people do for Travis typically. [00:51:43] Saves resources. [00:51:59] And complexity. And developer sanity by having only one entry point when testing locally. [00:52:50] https://github.com/Krinkle/intuition/blob/master/composer.json#L25 [00:52:58] Anyway, let's roll with current convention for now. [00:53:39] Krinkle it saves resources but both run the wrong php version [00:53:50] composer now runs on nodepool, same for npm. [00:53:56] composer is trusty [00:54:00] and npm is jessie [00:54:01] https://gerrit.wikimedia.org/r/#/c/303116/ [00:54:12] one runs php 5.5 and the other 5.6 [00:54:14] thanks [00:54:27] permissions [00:54:30] James_F you need to press publish [00:54:36] please [00:54:41] Done. [00:54:48] Thanks [00:54:50] (Grumble grumble.) [00:55:32] So deploy the CI change, recheck then merge https://gerrit.wikimedia.org/r/#/c/303116/ , then recheck and merge https://gerrit.wikimedia.org/r/#/c/301301/ [00:55:46] You can do check experimental [00:56:08] (03Abandoned) 10Krinkle: Remove unused 'npm' job for ParserMigration [integration/config] - 10https://gerrit.wikimedia.org/r/303115 (owner: 10Krinkle) [00:56:19] No need. [00:56:23] It can be merged as-is. [00:56:29] As long as the package.json patch is merged first. [00:56:29] * paladox goes back to watching tv, 2am here. :) [00:58:11] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2525383 (10Legoktm) Do we have a core dump? [01:29:50] * paladox hates it when you get to a good part of a show and then you have to wait a year before you get to find out what happends [04:38:19] 06Release-Engineering-Team, 10Phabricator: Requesting access for @Danny_B to Phame - https://phabricator.wikimedia.org/T142142#2524197 (10Legoktm) Entirely off topic, but 142142! [05:00:05] PROBLEM - Puppet staleness on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:49:42] Project mediawiki-core-code-coverage build #2180: 04FAILURE in 2 hr 49 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2180/ [06:02:58] PROBLEM - Puppet staleness on deployment-eventlogging04 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [08:38:45] 10Deployment-Systems, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics-Cluster, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2526099 (10elukey) Updated https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Refinery We have some issues currently with th... [08:55:25] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526134 (10phuedx) [303083](https://gerrit.wikimedia.org/r/303083) a... [09:22:10] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526146 (10phuedx) [303090](https://gerrit.wikimedia.org/r/#/c/30309... [09:23:30] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526148 (10phuedx) [09:29:07] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526152 (10phuedx) >>! In T142141#2526134, @phuedx wrote: > [303083]... [09:36:50] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526166 (10phuedx) [09:38:35] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524100 (10phuedx) [09:38:56] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526171 (10phuedx) [09:42:12] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2526193 (10phuedx) @Jdlrobson, @bmansurov, @jhobs: 1? [10:43:10] ping grrrit-wm [10:57:43] 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, 10VisualEditor: Merge service-specific instances in deployment-prep to the deployment-sca* instances - https://phabricator.wikimedia.org/T142100#2526381 (10mobrovac) >>! In T142100#2524532, @greg wrote: > * Citoid: Crea... [11:04:28] 10Continuous-Integration-Infrastructure, 06Operations, 07Blocked-on-Operations, 07Zuul: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2526415 (10elukey) @hashar there is a problem uploading the jessie package: ``` Unable to find pool/thirdparty/z/zuu... [11:09:35] Interesting regression with page editing in https://phabricator.wikimedia.org/T142137 , making textarea content disappear. Wondering who to CC. [11:11:28] andre__ is that VisualEditor [11:11:38] Since it says WikiEditor was not a problem. [11:16:56] paladox, where does it say that? [11:17:23] here there is no problem if wikEd is turned on (with visible syntaxhighlight) [11:17:50] paladox, so where does it say that WikiEditor is not a problem? [11:18:00] On the task description [11:18:14] paladox, in which sentence? [11:18:44] in dewiki since about one hour (~ 2016-08-04T21:50:00 MESC ): [11:18:44] When I try to edit a page or a section the textarea get empty [11:18:44] the problem is reproducibel but not in every edit visible. [11:18:44] there is no problem if wikEd is turned on (with visible syntaxhighlight) [11:18:44] the problem is reproduceable at chrome and firefox. [11:18:45] Please unbrake this issue now! [11:19:14] paladox, yes, you just posted the complete task description. [11:19:17] I'm aware of it. [11:19:20] Yep [11:19:21] My quewstion however was: [11:19:23] paladox, so where does it say that WikiEditor is not a problem? [11:19:36] Im not sure [11:19:39]  [11:20:00] if you are not sure then why did you write it? [11:20:06] I don't get this conversation. [11:20:42] paladox: As far as I know, noone has ever written that WikiEditor is not a problem. You state differently. I asked you to explain why. [11:20:48] You said you doint know who to cc, i was asking was it visualeditor [11:21:05] paladox, you said "Since it says WikiEditor was not a problem" [11:21:06] wikEd is wikieditor [11:21:10] No. [11:21:34] Ok [11:21:40] wikEd is not WikiEditor. [11:22:35] Oh, sorry i thought it was, sorry [12:01:43] Yippee, build fixed! [12:01:43] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #103: 09FIXED in 42 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/103/ [12:01:55] Yippee, build fixed! [12:01:56] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #103: 09FIXED in 54 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/103/ [12:34:17] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2526762 (10Pokefan95) When... [13:50:40] ostriches sorry for the pin but the disscussion we had yesturday that i have to be user gerrit2 to get it to work [13:50:46] how can i run puppet as the user [13:50:52] since puppet applys the gerrit deb [13:51:06] but it fails for me running as user gerrit2 but works as root [13:51:31] You have to run puppet as root. You run gerrit.war as gerrit2. [13:53:03] ostriches, oh ah, why not just add a script that will make all the files gerrit2 in /var/lib/gerrit2/ [13:53:17] which then as root you can run puppet which will run all the scripts [13:53:40] for http i have https://gerrit.wikimedia.org/r/#/c/303146/ [13:55:07] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:56:22] paladox: Because nothing should be owned root in the folder anyway, except the plugin and war file that the package installs. [13:56:33] Puppet should make the rest of the files as gerrit2. If it doesn't, we can fix that. [13:56:38] I mean allow root to run puppet [13:56:44] but betend that it is gerrit2 [13:56:55] betend = pretends [13:57:12] Nope it dosent make the files gerrit2 i doint think either [13:57:17] most of them said root [13:57:25] That's wrong, they should be owned by gerrit2. [13:57:28] It's in the manifest. [13:57:39] Oh, but because puppet is run by root [13:57:42] the deb is installed [13:57:47] and the files are then root [13:57:57] So some of the files are gerrit2 [13:58:00] some arnt [13:58:53] The only files that the deb installs are the plugins and the war. [13:59:01] Which is ok to be root, as long as they're readable by gerrit2. [13:59:03] Oh yeh [13:59:09] o.O [13:59:14] 511 new phab notifications [13:59:22] since yesterday 21:30 UTC [13:59:25] LOL [13:59:34] Luke081515: Somebody's popular :p [13:59:42] i guess root gets added to permission [13:59:46] and gerrit2 dosent [13:59:50] oh yes now i know why [14:00:02] ostriches: I have more than 100 notifications of "Jdforrester-WMF lowered the priority of" [14:00:19] Hehe, I set priority to be ignored notification :p [14:00:19] about 200 I guess [14:00:42] But what i doint get is it seems to run init partially [14:00:50] but not fully so we doint get the files in bin/ [14:01:00] but it makes the files be owned by root [14:01:07] some are gerrit2 and some are root [14:01:29] Because the init script is run because puppet is run it uses root [14:01:32] as the user [14:01:44] If init fails, that's a problem. We have to figure out why. [14:01:56] No, the init script is run by gerrit2. [14:02:03] It's in puppet and says so [14:02:16] Oh [14:02:22] Im not sure why it dosent run then [14:02:27] James_F|Away: You got a bit hyperactive this night, heh? :D [14:02:42] since it keeps failing at reindex. [14:02:54] exec { 'install_gerrit_jetty': [14:02:54] creates => '/var/lib/gerrit2/review_site/bin', [14:02:54] user => 'gerrit2', [14:02:54] group => 'gerrit2', [14:02:54] cwd => '/var/lib/gerrit2', [14:02:55] command => '/usr/bin/java -jar gerrit.war init -d review_site --batch --no-auto-start', [14:02:55] require => [ [14:02:56] File['/var/lib/gerrit2/review_site/etc/gerrit.config'], [14:02:56] File['/var/lib/gerrit2/review_site/etc/secure.config'], [14:02:57] ], [14:02:57] } [14:03:00] See, user/group gerrit2. [14:03:13] Yep [14:03:15] hm, AND I have about 50 notifications of Nemo_bis, reverting James_F|Away [14:03:17] Same thing with reindex. [14:03:30] But it dosent seem that is run for me, it runs reindex, unless because i run puppet again [14:03:34] after it fails the first time [14:03:56] Well you gotta figure out why it fails :) [14:04:58] Ok [14:04:59] yep [14:05:07] Could you review [14:05:08] https://gerrit.wikimedia.org/r/#/c/303146/ [14:05:09] please? [14:05:23] I hope you like the way i set it out [14:05:33] reduces alot of code duplication. [14:06:23] Oh yes sorry i shoulden have ask you to review [14:06:28] since you get it in your email [14:06:31] sorry about that [14:07:05] * paladox will try the instance creation again. [14:07:54] I'll have a look later. [14:08:01] Ok [14:08:07] Also i found some problems with ssh [14:08:10] on gerrit.wikimedia.org [14:08:24] mediawiki-config was failing last night [14:08:30] it had connection timeout [14:08:39] but once i re downloaded it through ssh it worked [14:08:43] so it seems strange [14:10:19] I prefer https :p [14:11:11] Oh [14:11:17] Isent that insecure? [14:11:23] Oh wait sorry [14:11:25] wrong http [14:11:53] Yeh, but letsencrypt will need to add a paid plan that lets you do unlimited certificutes without the rate limit [14:12:48] I'm sure we can work something out with them without paying :) [14:13:39] Oh [14:13:41] yeh [14:13:55] Guessing they wont reject the world largest wikipedia group [14:13:56] :) [14:15:21] But we should use that patch [14:15:25] until they do [14:15:44] and then revert it when we work something out with them without paying :) [14:16:51] One thing though maybe init is run but it deffitly does not create gerrit.war in bin/ [14:16:57] unless i manually run it [14:17:14] i get this error [14:17:15] Notice: /Stage[main]/Gerrit::Jetty/Exec[install_gerrit_jetty]/returns: Exception in thread "main" java.lang.RuntimeException: Cannot save secure.config [14:17:15] Notice: /Stage[main]/Gerrit::Jetty/Exec[install_gerrit_jetty]/returns: at com.google.gerrit.server.securestore.DefaultSecureStore.save(DefaultSecureStore.java:88) [14:17:27] But the file saved [14:18:52] Notice: /Stage[main]/Gerrit::Jetty/Exec[install_gerrit_jetty]/returns: Caused by: java.io.IOException: Permission denied [14:19:13] Yeh init is run [14:19:14] but fails [14:19:15] Error: /usr/bin/java -jar gerrit.war init -d review_site --batch --no-auto-start returned 1 instead of one of [0] [14:19:15] Error: /Stage[main]/Gerrit::Jetty/Exec[install_gerrit_jetty]/returns: change from notrun to 0 failed: /usr/bin/java -jar gerrit.war init -d review_site --batch --no-auto-start returned 1 instead of one of [0] [14:23:08] Hmmmmm [14:23:28] Ah ok, I see it now. [14:23:50] /var/lib/gerrit2/review_site gets created by the package as root, so the init script can't write the bin. [14:24:09] Same thing with index, probably. [14:24:35] Oh [14:24:48] How can i get it to work as gerrit2? [14:24:55] Hmmm. [14:25:08] But puppet should ensure on that directory, makes sure it's the right permissions. [14:25:16] Ok let me check [14:25:34] Yep it has correct permission [14:25:36] gerrit2 [14:25:59] But root should be able to access everything [14:29:54] It saves [14:29:55] Notice: /Stage[main]/Gerrit::Jetty/File[/var/lib/gerrit2/review_site/etc/gerrit.config]/ensure: defined content as '{md5}edb5b24f5327a1f243fda23e5d8c8bef' [14:29:57] correctly [14:30:15] Maybe because of the permissions https://github.com/wikimedia/operations-puppet/blob/9cbcec96036b3a5d34acd56c1cabaee589822e46/modules/gerrit/manifests/jetty.pp#L82 [14:30:27] https://github.com/wikimedia/operations-puppet/blob/9cbcec96036b3a5d34acd56c1cabaee589822e46/modules/gerrit/manifests/jetty.pp#L90 [14:33:11] Is there a way to shut notifications from "phabricator_maintance" down? [14:33:31] * Luke081515 got 124 notifications again [14:34:24] Im not sure [14:36:07] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2527357 (10bmansurov) >>! In T142141#2526134, @phuedx wrote: > [3030... [14:37:10] Should it even try to save secure.config [14:37:13] in gerrit [14:37:20] i doint think so so strange. [14:37:57] Ah [14:37:59] Notice: /Stage[main]/Gerrit::Jetty/File[/var/lib/gerrit2/review_site/etc/secure.config]/ensure: defined content as '{md5}77a34300b13abc58f3ac14893e5983c6' [14:38:16] it saves it but then gerrit trys to overinstall secure.config but it carnt [14:38:31] so it fails causing init to fail since any error in java fails the whole thing [14:38:48] Maybe we will want to leave secure.config saving until the end? [14:43:43] Maybe we could move https://github.com/wikimedia/operations-puppet/blob/9cbcec96036b3a5d34acd56c1cabaee589822e46/modules/gerrit/manifests/jetty.pp#L129 [14:43:45] to reindex [14:43:53] ? [14:44:30] but also move the one gerrit downlaods into a new name or symblink it to a custom named secure.conf file [14:55:05] RECOVERY - Free space - all mounts on deployment-eventlogging03 is OK: OK: All targets OK [14:58:30] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:58:32] 10Deployment-Systems, 06Operations: error on tin:/srv/mediawiki-staging: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T127093#2527467 (10Dzahn) Yep, that. I think it happens so regularly now, and we never see a root actually doing stuff ar... [14:59:16] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:05:06] RECOVERY - Puppet staleness on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:07:58] RECOVERY - Puppet staleness on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:10:16] I have this patch [15:10:16] https://gerrit.wikimedia.org/r/#/c/303173/ [15:10:20] not sure if it is correct [15:10:41] but really we need to do secure.config after gerrit installs due to it trying to install its own secure.config file [15:26:46] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2527538 (10phuedx) [15:28:01] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2524100 (10phuedx) @Jdlrobson, @jhobs: Update the point value if you... [15:57:56] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:08:16] 10Deployment-Systems, 06Operations: error on tin:/srv/mediawiki-staging: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T127093#2527653 (10demon) >>! In T127093#2527467, @Dzahn wrote: > Yep, that. I think it happens so regularly now, and we... [16:22:27] I'm having a strange failure (segfault) in https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6250/console [16:22:43] related patch is small one line fix: https://gerrit.wikimedia.org/r/#/c/303178/ [16:22:54] dcausse https://phabricator.wikimedia.org/T142158 [16:23:10] paladox: thanks, sorry for the noise [16:23:34] your welcome, plus i doint think it is noise, since you didnt know. [16:56:15] ostriches setting passwords::gerrit::gerrit_db_pass [16:56:17] does not work [16:56:32] and for https://gerrit.wikimedia.org/r/#/c/303173/ [16:56:38] i renamed the secure.config file [16:57:02] since gerrit when it runs it wants to generate secure.config [16:57:07] Well puppet knows nothing about secure2.config, so the symlink will just be broken. [16:57:18] Yeh i have renamed the file [16:57:27] See https://gerrit.wikimedia.org/r/#/c/303173/5/modules/gerrit/templates/secure2.config.erb [16:58:07] Looser permissions on the files would be easier. [16:58:18] I want it so that we can wipe out the secure.config file after we install gerrit without one. [16:58:18] Than symlinks. [16:58:19] and oh [16:58:36] Im not sure how to do that, doing that in puppet was a guess [16:58:45] since i doint know fully how puppet works [16:59:04] Well it's in the same file :) [16:59:07] We'll just change the permissions from 0444 to 0664 [16:59:13] Oh ah [16:59:16] will that work? [16:59:42] but then wont gerrit overide our secure.config file [17:00:00] since it will be installed before gerrit installs [17:00:16] i guess we should move secure.config to reindex step? [17:00:52] gerrit will override it, but our next puppet run will restore it ;-) [17:00:57] Oh ah [17:01:01] Plus, the writes it usually wants to do is just fixing whitespace, etc. [17:01:07] oh [17:01:12] It's just cleaning up existing config 99% of the time [17:01:30] The way "gerrit" thinks the file should look ;-) [17:01:31] lol [17:02:11] https://gerrit.wikimedia.org/r/#/c/303203/ [17:02:17] https://gerrit.wikimedia.org/r/#/c/303206/ [17:02:17] Oh [17:02:30] LOL [17:02:41] We'll do mine, it does the same for all 3 .config files [17:02:45] Yep [17:02:51] Ive abandoned mine :) [17:03:04] For the db password how do i set that please? [17:03:58] I set the password here https://wikitech.wikimedia.org/w/index.php?title=Hiera:Git&action=edit [17:04:05] but it dosent seem to have changed anything [17:04:09] even when running puppet [17:04:25] You've got a stray comma at the end, for one [17:04:34] Oh [17:04:42] Woop [17:04:44] wopps [17:04:46] woops [17:05:15] You could set it in the labs/private repo. [17:05:23] The other option is just reusing the password already in labs/private :p [17:05:28] (that's what I did. I was lazy) [17:05:38] Oh [17:05:43] Where do i find that repo [17:05:44] ? [17:05:58] please [17:06:00] https://phabricator.wikimedia.org/diffusion/LPRI/browse/master/modules/passwords/manifests/init.pp;HEAD$154 [17:06:03] Thanks [17:06:06] Repo is called labs/private :) [17:06:26] LOL [17:06:33] that is the password it gave me [17:07:32] * paladox is waiting for https://gerrit.wikimedia.org/r/#/c/303206/ to be merged :) [17:09:18] * paladox loves the new inline edit gerrit, needs high priority for phabricator too [17:22:18] (03PS1) 10BryanDavis: labs/striker: Add test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/303218 [17:27:06] (03PS2) 10BryanDavis: labs/striker: Add test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/303218 [17:44:12] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-UserMerge: Error: 1137 Can't reopen table: 'w1' - https://phabricator.wikimedia.org/T142240#2528200 (10Reedy) [17:45:11] PROBLEM - Puppet run on deployment-db01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:49:54] 06Release-Engineering-Team, 15User-greg: Create agenda outline for 2016 RelEng team offsite - https://phabricator.wikimedia.org/T138437#2528266 (10greg) [17:50:16] !log Removed stale cherry-picks for https://gerrit.wikimedia.org/r/#/c/302303/ and https://gerrit.wikimedia.org/r/#/c/300458/ that were blocking git rebase [17:50:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:54:02] !log Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/3 for testing [17:54:02] Project mediawiki-core-code-coverage build #2181: 04STILL FAILING in 2 hr 54 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2181/ [17:54:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:56:10] Project beta-scap-eqiad build #114243: 04FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114243/ [18:03:07] greg-g: Hey. Just a heads-up that an interesting recent regression with page editing (and probably some gadget) in https://phabricator.wikimedia.org/T142137 gets quite some attention on Village Pumps. Makes the textarea content disappear. And I am not sure who to CC. :-/ [18:06:16] Project beta-scap-eqiad build #114244: 04STILL FAILING in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114244/ [18:08:32] "20:06:16 18:06:16 Check 'Logstash Error rate for deployment-mediawiki01.deployment-prep.eqiad.wmflabs' failed: WARNING: Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /logstash-*/_search" [18:08:47] 20:06:16 18:06:16 Unhandled error: [18:08:47] 20:06:16 Traceback (most recent call last): [18:08:48] 20:06:16 File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 255, in run [18:08:53] meh :-/ [18:09:43] andre__, can you get Boshomi to disable gadgets too? [18:10:22] if you want to reach him, I think the best would be to write at his talk page @ dewiki [18:12:47] Luke081515: that scap failure may have been caused by me changing logstash config in beta [18:13:18] although... that shouldn't have restarted Elasticsearch [18:16:23] Project beta-scap-eqiad build #114245: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114245/ [18:17:17] blerg. that's very weird. I can run logstash_checker.py manually just fine, seemingly :\ [18:22:03] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.056 second response time [18:26:34] Yippee, build fixed! [18:26:35] Project beta-scap-eqiad build #114246: 09FIXED in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114246/ [18:26:49] hmm, looks like it was hitting port 80 on the logstash server in the scap.cfg file for eqiad.wmflabs :\ [18:26:58] how did that work before? [18:27:16] it would have worked before we switched to kibana4 [18:27:40] the nginx proxy used to pass through to _search [18:28:01] now kibana4 gets that traffic and probably barfs [18:28:51] yeah, but that didn't change today on beta? I had to rewrite that whole logstash_checker query for new logstash stuff a couple weeks ago. [18:29:41] *nod* One thing that changed was my unbreaking puppet replication ~40 minutes ago [18:30:06] so maybe something that was updated for prod is now not quite right in beta cluster? [18:30:13] hmm. I wonder if the patch I got merged yesterday was an iteration that I didn't have cherry-picked on beta originally. [18:30:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.079 second response time [18:30:34] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.101 second response time [18:36:13] Project beta-scap-eqiad build #114247: 04FAILURE in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114247/ [18:38:00] Invalid callback Timeline::onParserFirstCallInit in hooks for ParserFirstCallInit [18:38:50] https://gerrit.wikimedia.org/r/301211 was merged just now [18:39:24] +$wgAutoloadClassp['Timeline'] = __DIR__ . '/Timeline.body.php'; [18:40:16] Reedy, FlorianSW [18:40:28] seems like a likely culprit for that error. [18:41:39] fixed it [18:41:41] will submit [18:42:04] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 44529 bytes in 2.583 second response time [18:43:38] Krenair was probaly missed since it needs jenkins to find the correct main file [18:43:38] to test [18:43:54] Krenair: https://gerrit.wikimedia.org/r/#/c/303241/ [18:44:00] I can submit a patch that will re add this support by using symblinks but this will not work on windows [18:44:10] damn, Krenair, I haven't read your messgae :( [18:44:13] I already submitted it FlorianSW [18:44:56] let's take yours :) [18:45:01] sorry for that :( [18:45:22] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 44522 bytes in 2.152 second response time [18:45:34] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44529 bytes in 1.510 second response time [18:46:28] Krenair https://gerrit.wikimedia.org/r/#/c/303243/ [18:46:31] Yippee, build fixed! [18:46:31] Project beta-scap-eqiad build #114248: 09FIXED in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114248/ [18:48:02] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.057 second response time [18:51:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.076 second response time [18:51:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2854 bytes in 0.081 second response time [18:53:23] for some reason it wiped my commit off of staging [18:53:35] as in, mediawiki-stagng [18:53:36] staging* [18:56:35] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44529 bytes in 1.159 second response time [18:57:08] ah, beta-code-update-eqiad probably did that. When it ran git submodule update --init --recursive [18:58:03] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 44527 bytes in 0.906 second response time [19:01:15] yeah that'd do it [19:01:24] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 44529 bytes in 1.756 second response time [20:29:06] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2528794 (10greg) [20:29:24] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2528806 (10greg) [20:29:42] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2528794 (10greg) [20:29:59] 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2528810 (10greg) [20:30:26] 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: Move citoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142150#2528811 (10greg) [20:40:37] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Enable CI jobs on the new CiviCRM repos - https://phabricator.wikimedia.org/T118604#2528846 (10DStrine) [20:40:40] 10Continuous-Integration-Config, 10Fundraising-Backlog, 07Technical-Debt: Use the name "grunt-jscs" in all Fundraising CI glue - https://phabricator.wikimedia.org/T115642#2528850 (10DStrine) [20:45:15] Project beta-scap-eqiad build #114260: 04FAILURE in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114260/ [20:46:47] 06Release-Engineering-Team, 15User-greg: Rework the #Beta-Cluster-Infrastructure workboard - https://phabricator.wikimedia.org/T138887#2528955 (10greg) [20:47:00] > Timeline/extension.json does not exist! [20:49:37] 10Continuous-Integration-Config, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 13Patch-For-Review: mediawiki/core fundraising/REL branches should use git submodule - https://phabricator.wikimedia.org/T100637#2529014 (10DStrine) [20:50:30] 07Browser-Tests, 10Continuous-Integration-Infrastructure, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Create unit and integration tests for Fundraising extensions to identify breaking MediaWiki changes - https://phabricator.wikimedia.org/T89404#2529046 (10DStrine) [20:53:17] thcipriani https://gerrit.wikimedia.org/r/#/c/303248/ [20:54:34] Reedy ^^ [20:54:53] 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: On beta cluster citoid should self update and reload after change is merged - https://phabricator.wikimedia.org/T95652#2529179 (10greg) [20:55:02] Project beta-scap-eqiad build #114261: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114261/ [20:59:34] oh, typo [21:02:22] Project beta-scap-eqiad build #114262: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114262/ [21:02:26] behave wmf-insecte [21:02:42] 06Release-Engineering-Team, 07Tracking, 15User-greg: Redo some #RelEng -related project workboard columns (tracking) - https://phabricator.wikimedia.org/T138884#2529261 (10greg) [21:02:44] 06Release-Engineering-Team, 15User-greg: Rework the #Beta-Cluster-Infrastructure workboard - https://phabricator.wikimedia.org/T138887#2529258 (10greg) 05Open>03Resolved a:03greg {{done}} except for the CI/CD column which may come later. Sorry for the spam. [21:03:40] 10Beta-Cluster-Infrastructure, 10Shinken: Shinken alert for beta error rate - https://phabricator.wikimedia.org/T141785#2529263 (10greg) p:05Triage>03High [21:04:43] Reedy: is that a new command on toollabs? like "become" we have "behave" now? [21:04:53] lol [21:04:59] Project beta-scap-eqiad build #114263: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114263/ [21:05:11] Yippee, build fixed! [21:05:11] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #108: 09FIXED in 4 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/108/ [21:05:14] lol [21:05:24] ^ that failure should fix in a minute or 2 [21:06:25] thanks for the restart [21:06:38] Your welcome [21:12:23] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529298 (10greg) p:05Tria... [21:12:51] Yippee, build fixed! [21:12:52] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #97: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/97/ [21:17:23] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529314 (10Tgr) >>! In T14... [21:21:31] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529336 (10greg) :) yep, s... [21:22:26] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529344 (10Tgr) Similar (b... [21:24:57] (03PS1) 10Paladox: [timeline] Add test extensions-unittests-generic [integration/config] - 10https://gerrit.wikimedia.org/r/303305 [21:25:18] (03PS2) 10Paladox: [timeline] Add test extensions-unittests-generic [integration/config] - 10https://gerrit.wikimedia.org/r/303305 [21:25:30] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529355 (10Tgr) Non-mobile... [21:25:58] Yippee, build fixed! [21:25:59] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #97: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/97/ [21:29:10] tgr: I swear to you I could repro without mobile before, but now I can't :/ [21:29:24] I'll let you/others figure it out [21:29:27] :) [21:31:34] Yippee, build fixed! [21:31:34] Project beta-scap-eqiad build #114264: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114264/ [21:31:47] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:39] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:43] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:49:42] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2529420 (10Tgr) Some small... [22:24:13] greg-g it worked [22:24:25] what did? [22:24:25] ostriches got it to work fully from puppet it seems. [22:24:34] gerrit installing through puppet [22:24:42] good deal! [22:24:47] Yep [22:24:52] just http will need fixing now [22:25:00] due to it hit the rate limit [22:25:08] yea, pretty cool! after some small fixes re: permissions on the config file [22:25:16] Yep [22:30:54] [2016-08-05 22:25:20,208] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.12.2 ready [22:31:14] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #98: 04FAILURE in 9 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/98/ [22:39:38] much nicer list https://phabricator.wikimedia.org/P3637 now :) [22:48:48] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2529539 (10Jdlrobson) All are passing except https://integration.wik... [23:07:43] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:38] greg-g: I'll deploy https://gerrit.wikimedia.org/r/#/c/303320/ , pywikibot was broken by that change so we need some time to get people to update their bots [23:11:38] tgr: if you feel it's warranted, sure thing [23:12:37] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44522 bytes in 2.004 second response time