[00:06:32] 10Continuous-Integration-Infrastructure, 10Scap (Scap3-Adoption-Phase1), 13Patch-For-Review: Migrate Zuul deployment of integration/config to scap deploy - https://phabricator.wikimedia.org/T129357#3086237 (10demon) [00:06:34] 10Scap, 10scap2: scap to reload a service instead of restart - https://phabricator.wikimedia.org/T134001#3086235 (10demon) 05Open>03Resolved [00:15:54] I get this error [00:15:54] java.lang.IllegalStateException: Missing project All-Projects [00:15:55] now [00:15:59] RainbowSprinkles ^^ [00:16:47] Hmm, it should auto-create that [00:17:20] Not if the db already has it [00:17:29] i will just drop the db and recreate it [00:17:45] Per this bug [00:17:46] https://bugs.chromium.org/p/gerrit/issues/detail?id=1305&can=2&start=0&num=100&q=&colspec=ID%20Type%20Stars%20Milestone%20Status%20Priority%20Owner%20Summary&groupby=&sort= [00:28:31] 10Beta-Cluster-Infrastructure, 10Flow, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), 13Patch-For-Review, 07Performance: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#850716 (10Jdforrester-WMF) Is this still an is... [00:34:29] Oh yeah I remember that bug [00:35:26] RainbowSprinkles it seems running java -jar gerrit.war init -d review_site makes review_site some of the files in there owned by gerrit2:nda [00:35:46] we should get puppet to change that so that it recursivly makes it gerrit2:gerrit2 [00:37:35] Well, other than config most of that stuff should be owned by gerrit2 [00:37:59] Yep [00:40:33] -rw-r--r-- 1 gerrit2 nda 75 Mar 9 00:39 gerrit_index.config [00:40:38] It keeps doing that [00:40:42] causing index to fail [00:41:21] Sounds like the gerrit2 umask is wrong on the labs instances [00:41:34] Or its default group is, rather [00:41:38] * RainbowSprinkles hmmmmms [00:41:46] yes [00:41:48] it seems it is wrong [00:41:58] How can i check [00:42:30] In ldap I guess [00:42:35] oh [00:42:45] I thought the user is created through the gerrit package [00:42:48] or puppet [00:42:53] I got rid of that I thought [00:43:16] Ah no I didn't [00:43:20] I guess /etc/passwd [00:43:24] Should list default group [00:43:36] "phd" phab service is now also using base::service_unit [00:43:37] In which case: why did the package create a gerrit2 in the nda group? [00:44:20] Not sure [00:45:08] theres no gerrit2 in /etc/passwd [00:45:13] though the group exists [00:45:17] paladox: /etc/group ? [00:45:21] as doing chown -R gerrit2:gerrit2 works [00:45:25] * paladox checks [00:46:11] searching with nano points at debug2: channel 0: window 999052 sent adjust 49524 [00:46:29] but yes [00:46:30] it exists [00:46:31] gerrit2:x:120: [00:47:06] https://phabricator.wikimedia.org/diffusion/ODDX/browse/master/debian/gerrit.postrm [00:47:25] https://phabricator.wikimedia.org/diffusion/ODDX/browse/master/debian/gerrit.postinst [00:49:23] root@gerrit-test3:/var/lib/gerrit2/review_site# groups gerrit2 [00:49:23] gerrit2 : nda labsadminbots [00:49:28] RainbowSprinkles ^^ [00:50:02] Grrrr, we really need to kill that godforsaken ldap account [00:50:04] It fights [00:50:09] * RainbowSprinkles rages [00:50:20] It's using the gerrit2 ldap account instead of a system one [00:50:32] lol [00:50:44] I think....we can [00:50:50] I don't think anything uses it anymore.... [00:50:58] Yep [00:54:13] :) [00:55:14] yea, this is definitely not the first time we have trouble with LDAP users conflicting with system users [00:55:33] other roles too afair [00:57:25] yep [00:57:27] oops, i see an issue with phab search now [00:57:45] * paladox wonders how the user is deleted [00:57:46] on ldap [00:57:50] [cURL/6] (phabricator/DREV,CMIT,ATOM,BOOK,CEVT,FITV,MOCK,OPKG,CDTL,BLOG,POST,WIKI,QUES,PROJ,TASK,USER/_search) There was an error resolving the server hostname. Check that you are connected to the internet and that DNS is correctly configured. (Did you add the domain to `/etc/hosts` on some other machine, but not this one?) [00:57:59] lol [00:57:59] can you confirm that? [00:58:08] yes thats pending a phab update [00:58:11] twentyafterfour: i saw the above when searching things [00:58:16] due to the new cluster search config [00:58:28] well, upgrade window starts in 2 min [00:58:33] mutante: yeah it'll be fixed [00:58:39] ok, all cool then :) [00:58:50] I had temp-fixed in in local.config but puppet re-broke it ;) [00:58:58] aha, gotcha [00:59:06] :) [00:59:56] ok, and with that i will stop doing anything on phab servers and let you do your planned upgrade [01:00:03] the service conversion is done [01:01:24] me logs out and gets a bus. if you need a root, text me at the number on office wiki and i can get back anytime with mobile hotspot [01:20:43] (03PS1) 10Krinkle: Ensure *-npm-node-6-jessie jobs get global-set-env for CHROME_BIN [integration/config] - 10https://gerrit.wikimedia.org/r/341954 [01:25:48] (03CR) 10Krinkle: "Affected: 6 jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/341954 (owner: 10Krinkle) [01:29:16] (03CR) 10Krinkle: [C: 032] Ensure *-npm-node-6-jessie jobs get global-set-env for CHROME_BIN [integration/config] - 10https://gerrit.wikimedia.org/r/341954 (owner: 10Krinkle) [01:30:44] (03Merged) 10jenkins-bot: Ensure *-npm-node-6-jessie jobs get global-set-env for CHROME_BIN [integration/config] - 10https://gerrit.wikimedia.org/r/341954 (owner: 10Krinkle) [02:05:03] 06Release-Engineering-Team, 10Elasticsearch, 13Patch-For-Review, 10Phabricator (Search): phab+elasticsearch: support multiple elasticsearch clusters / datacenters - https://phabricator.wikimedia.org/T157156#3086570 (10mmodell) 05Open>03Resolved [02:05:05] 06Release-Engineering-Team, 07Performance, 10Phabricator (Search): Extremely slow Phabricator search (1m 45s) - https://phabricator.wikimedia.org/T159575#3086572 (10mmodell) 05Open>03Resolved [04:56:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [05:19:23] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:25] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:02] 10Gerrit: Gerrit code review view jumps/scrolls up and down when commenting - https://phabricator.wikimedia.org/T159919#3086794 (10santhosh) Can't reply to CR comments too. It just scrolls up and you can't focus on text area to type the reply [07:16:33] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:23:22] 10Gerrit: Gerrit code review view jumps/scrolls up and down when commenting - https://phabricator.wikimedia.org/T159919#3086977 (10Paladox) I believe this is a codemirror issue inside of gerrit but I may be wrong. This should still be reported against gerrit. Anyways could someone try https://gerrit-review.goo... [09:24:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: zuul-merger git-daemon process is not started properly by systemd ? - https://phabricator.wikimedia.org/T157785#3087100 (10hashar) From my comment on the hack https://gerrit.wikimedia.org/r/#/c/340496/ The `git-daemon-s... [09:33:35] zeljkof: the selenium nodejs patch pass all fine https://integration.wikimedia.org/ci/job/mediawiki-core-selenium-jessie/76/console [09:33:38] and jon replied on it :-} [09:33:48] wonder why there is no color output in jenkins though :-(((( [09:34:05] there are bunch of blank lines as well. Seems something console.log("\n") [09:34:11] hashar: nice! [09:34:28] I saw another comment on the patch, from somebody else [09:35:05] yeah millimetric [09:39:52] !log deployment-prep: rebasing puppet master. Got stall due to a submodule update apparently [09:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:48:09] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:50:57] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:01:10] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3087134 (10Tobi_WMDE_SW) After several days of stability, the last 2 builds had several strange failures again:... [10:03:03] Yippee, build fixed! [10:03:04] Project selenium-Core » firefox,beta,Linux,BrowserTests build #344: 09FIXED in 6 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/344/ [10:28:05] RECOVERY - Puppet run on deployment-phab02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:55] RECOVERY - Puppet run on deployment-phab01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:51:56] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3087375 (10Jhernandez) [13:38:03] hashar: whats on the agenda for today anything fun? [13:47:09] Yippee, build fixed! [13:47:10] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #331: 09FIXED in 3 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/331/ [14:04:16] okay im going crazy it is retest right? [14:13:23] Zppix: 'recheck' [14:13:34] hashar: can you slap me a few times with trout [14:23:03] PROBLEM - Puppet run on deployment-copper is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:25:30] build logs (on integration.wikimedia.org/ci/ ) went down for a second there hashar fyi [14:46:06] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:05:19] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3087905 (10hashar) + @demon for the release/build phase of the Popups extension. I am going to first derail the topic before providing an actu... [15:08:11] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3087930 (10hashar) Just to be more specific: ignore my rant about not using webpack. Lets focus on how we can: * get rid of those assets from t... [15:21:04] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:29] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088003 (10zeljkofilipin) p:05Normal>03Low [15:22:32] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Depool precise jenkins instances - https://phabricator.wikimedia.org/T158652#3088004 (10hashar) Will do that next week. I have announced it on wikitech-l: ``` Subject: CI dropping support for Zend PHP 5.3 Hello, Next w... [15:25:31] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088010 (10zeljkofilipin) [15:26:30] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 10Wikidata, 15User-zeljkofilipin: Run Wikibase browser tests on gerrit triggered with keyword - https://phabricator.wikimedia.org/T145190#3088013 (10Tobi_WMDE_SW) [15:26:33] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 13Patch-For-Review, 07Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#3088012 (10Tobi_WMDE_SW) [15:26:47] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088015 (10Jhernandez) > Would it be sufficient to add globing support to extension registry? The problem with this would be script order. File... [15:27:14] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025671 (10zeljkofilipin) [15:27:47] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 10Wikidata, 15User-zeljkofilipin: Trigger run of special set of browsertests on gerrit with a keyword - https://phabricator.wikimedia.org/T145190#2622727 (10Tobi_WMDE_SW) [15:33:54] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088026 (10zeljkofilipin) [15:35:15] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088028 (10Jhernandez) @hashar As an extension having a separate branch for deployment seems quite controversial. I don't have any preferences t... [15:35:33] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025671 (10zeljkofilipin) [15:39:05] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088048 (10zeljkofilipin) [15:42:06] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088054 (10zeljkofilipin) [15:42:48] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025671 (10zeljkofilipin) [15:44:17] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3088056 (10zeljkofilipin) [15:45:06] Yippee, build fixed! [15:45:07] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #353: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/353/ [15:48:35] 10Browser-Tests-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 07Ruby, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025671 (10zeljkofilipin) a:05zeljkofilipin>03None mediawiki_selenium Ruby gem is updated. mediawiki/core is updated. The majori... [15:52:04] 10Browser-Tests-Infrastructure, 07JavaScript, 06Maps (Kartographer), 13Patch-For-Review, 15User-zeljkofilipin: Set up Selenium tests for Kartographer extension - https://phabricator.wikimedia.org/T152683#3088097 (10zeljkofilipin) I have proposed a skill share session for Vienna hackathon {T159945} if you... [15:52:44] Yippee, build fixed! [15:52:45] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #353: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/353/ [16:03:08] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3088154 (10zeljkofilipin) @Tobi_WMDE_SW I am working on this. Looks like the only way to ensure stability is to r... [16:08:11] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088163 (10Jhernandez) We've chatted a bit on irc and there are jobs that require the compiled assets to be present, like the php ones that will... [16:09:16] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088164 (10hashar) >> Would it be sufficient to add globing support to extension registry? > The tooling is there for sound reasons, it allows u... [16:11:14] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088166 (10Paladox) [16:11:23] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088178 (10Paladox) p:05Triage>03Unbreak! [16:12:58] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088188 (10Dzahn) p:05Unbreak!>03High [16:13:31] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088166 (10Dzahn) lowered from UBN to High since search works for me and no user impact has been described. [16:15:31] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088193 (10Paladox) @Dzahn the bin/search init generates the index for elasticsearch. So if some how the index got delete... [16:17:48] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088200 (10Jhernandez) [16:20:53] !log upgrading elasticsearch on deployment-prep to v5.1.2 [16:20:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:30:12] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088214 (10zeljkofilipin) [16:30:35] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088229 (10zeljkofilipin) p:05Triage>03Normal [16:33:09] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3088234 (10Tobi_WMDE_SW) @zeljkofilipin Ok! Thanks for your effort and for the update! If there's anything we can... [16:35:17] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:40:20] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:43:57] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088253 (10zeljkofilipin) [16:45:07] (03PS3) 10Zfilipin: WIP Problem: Can not use --retry option to retry failed tests as part of the same run [selenium] - 10https://gerrit.wikimedia.org/r/341523 (https://phabricator.wikimedia.org/T160086) [16:46:29] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3088261 (10zeljkofilipin) [16:46:31] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088260 (10zeljkofilipin) [16:47:36] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:00:57] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088285 (10zeljkofilipin) Looks like the command line option is there in Cucumber 2.4, but the feature does not work. A simple test: `... [17:02:14] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 2 - https://phabricator.wikimedia.org/T160086#3088287 (10zeljkofilipin) Retry works as expected with the latest prerelease version of Cucumber (3.0.0.pre.1): ``` $ bundle exec cucu... [17:10:15] 06Release-Engineering-Team, 06Operations, 05DC-Switchover-Prep-Q3-2016-17: Understand the preparedness of misc services for datacenter switchover - https://phabricator.wikimedia.org/T156937#3088335 (10fgiunchedi) [17:15:18] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:20] RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:17:33] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:38] thcipriani: have you seen https://git-lfs.github.com ? [17:21:39] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 07Ruby, 15User-zeljkofilipin: Upgrade Cucumber from version 1 to version 3 - https://phabricator.wikimedia.org/T160086#3088379 (10zeljkofilipin) [17:21:40] urandom: I have seen it, I'm not super familiar with it. I know that it is in the same space as git fat and uses the same mechanism in git (smudge and clean) to restore large git files. Apart from that, I have no first-hand experience. [17:22:22] (03PS4) 10Zfilipin: WIP Problem: Can not use --retry option to retry failed tests as part of the same run [selenium] - 10https://gerrit.wikimedia.org/r/341523 (https://phabricator.wikimedia.org/T160086) [17:22:47] urandom: why do you ask? [17:23:51] thcipriani: just stumbled on it, and figured i mention it [17:24:03] thcipriani: since git-fat hasn't been entirely trouble-free :) [17:24:22] hehe thanks :) [17:25:29] yeah, we looked at that with scap, ultimately decided it'd be easier to transition from trebuchet if we kept the large file store the same; however, the one I was (am) really excited about is git-annex https://git-annex.branchable.com/ [17:26:46] thcipriani: what makes it exciting? [17:26:57] I use it for a lot of personal stuff and it's pretty magic. Plus it *doesn't* use git filters which is kinda nice https://git-annex.branchable.com/todo/smudge/ [17:27:22] the number and type of backends is pretty flexible, it's well thought out, seems pretty mature [17:27:32] neat [17:28:37] I use it personally for photography. I keep all my raw files in it, keep copies on my NAS and encrypted on S3. It's really cool. It's complex, but it seems solid. [17:29:08] you can also waste a day looking at all of its crazy features :) [17:29:27] https://git-annex.branchable.com/tips/peer_to_peer_network_with_tor/ [17:50:31] * greg-g uses git-annex for all things like Music, Photos, Documents, and a "hodge-podge" repo called just "annex" [17:53:24] * hashar has an Andrew FileSystem setup [18:36:12] It's so hard debugging ios js error's [18:36:26] I get undefined is not an object [18:37:39] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088698 (10Paladox) a:03mmodell [18:56:56] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088743 (10demon) >>! In T158980#3088028, @Jhernandez wrote: > @hashar As an extension having a separate branch for deployment seems quite contr... [18:58:48] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3088747 (10Jdlrobson) > Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets,... [19:15:01] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088808 (10Dzahn) @Paladox thanks for reporting it!. i think High is justified if you notice something like this after a... [19:15:10] * legoktm pokes RainbowSprinkles about https://phabricator.wikimedia.org/T156445 [19:16:16] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088810 (10Paladox) Oh yep. [19:31:49] we are shutting down fluorine. it's replaced by mwlog1001. technically "deployment-fluorine02.deployment.prep" is now an outdated name too [19:32:09] seeing that because it's the last remnant of "fluorine" in repoi [19:32:26] but i guess it's just names .. [19:45:36] RainbowSprinkles hi, did you manage to remove user gerrit2 from nda? [19:45:53] I didn't do anything with it, been busy [19:46:01] oh ok [19:46:01] legoktm: Ouch....stop poking :p [19:46:29] I guess we could use puppet to remove nda and add gerrit2 [19:46:41] RainbowSprinkles: mostly if you could say "these parts are hard" I'm happy to start working on them :) [19:48:07] So what I really want to do (and I haven't) is rewrite make-release in python instead of that ugly-ass PHP its in now, written in a way that it can be run daily on N branches and spit out tarballs. [19:48:24] *Also* taking a hidden directory of patches and spitting out some hidden tarballs [19:48:40] So we can constantly be *generating* the next security release, aware of conflicts, and only have to decide on a date and sign shit [19:49:06] So basically, "generate list of patches" will be "what patches has security dropped in the build pipeline?" [20:01:15] RainbowSprinkles just wondering if i uninstall gerrit would the group gerrit2 still exist? [20:01:45] even though it should be removed by https://phabricator.wikimedia.org/diffusion/ODDX/browse/master/debian/gerrit.postrm [20:03:38] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T158996#3088960 (10mmodell) [20:03:44] Removing user `gerrit2' ... [20:03:44] userdel: user gerrit2 is currently used by process 10677 [20:08:03] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088962 (10mmodell) Now to figure out how we would file an UBN bug when phabricator is down ;) [20:09:01] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3088975 (10Paladox) lol, use phab-01? [20:09:21] paladox: it's still running when you try to remove the user running it [20:09:29] Yep [20:10:15] so the postrm(?) script that removes the user, isnt making sure it is actually stopped/killed.. i guess [20:11:19] Yep [20:11:21] Let's be clear: that debian package is a horrible horrible hack [20:11:24] i found a script we could use. [20:11:34] It should not be depended on to do anything useful other than putting some files on a disk [20:11:43] I'm going to kill most of that debian package anyway [20:11:56] (eventually) [20:12:09] if (( $(ps -ef | grep -v grep | grep $service | wc -l) > 0 )) [20:12:37] 10MediaWiki-Releasing, 06Release-Engineering-Team, 06Security-Team: Streamline/automate MW tarball security release process - https://phabricator.wikimedia.org/T156445#3088982 (10Legoktm) ```lang=irc [11:48:07] So what I really want to do (and I haven't) is rewrite make-release in python i... [20:27:19] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3089053 (10Paladox) >>! In T160085#3088962, @mmodell wrote: > Now to figure out how we would file an UBN bug when phabric... [20:36:44] recommened "killall" but debian-lint says killall is dangerous :) ok [20:49:30] hey, any ideas why this is failing: https://gerrit.wikimedia.org/r/#/c/342073/1 ? [20:54:07] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3089211 (10hashar) > @hashar with regards to a MediaWiki standard, we are totally committed to that. Right now we are doing an experiment we fee... [21:03:40] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089287 (10Paladox) [21:03:44] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089299 (10Paladox) p:05Triage>03High [21:05:51] hashar: Hey, do you know how this is even possible? https://gerrit.wikimedia.org/r/#/c/342039/ (The tests pass but merge-test fails) [21:06:05] Amir1: an URL with numbers sure ? :} [21:06:42] the first thing Zuul does is try to merge the patch against tip of master. If that fails, no jobs are triggered. Zuul reports immediately [21:06:55] ah [21:07:08] the set of jobs is different [21:07:30] on patchset proposal we run mediawiki PHPUnit "extensions" suite under HHVM [21:07:34] that is the job mwext-testextension-hhvm-composer [21:07:48] on CR+2 we run that job AND the exact same tests but with Zend 5.5 [21:08:32] the later are failing. You can trigger solely the php5 job by commenting: check php5 [21:08:35] or 'check zend' [21:08:55] ruflin/elastica 5.1.0 requires php >=5.6.0 -> your PHP version (5.5.9) does not satisfy that requirement. [21:09:30] which most probably is a requirement added for the ElasticSearch 5 upgrade [21:09:39] which on wikimedia might work under HHVM [21:09:56] but would not with mw minimal requirement for PHP 5.5 [21:10:09] eg build / dependencies are broken [21:10:23] Amir1: ^^^ the long explanation [21:10:45] RainbowSprinkles i filled https://phabricator.wikimedia.org/T160122 for gerrit2 thingy [21:11:14] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3089320 (10Dzahn) >>! In T160085#3088962, @mmodell wrote: > Now to figure out how we would file an UBN bug when phabricat... [21:11:36] hashar: Thanks. This is really strange to me. Let me check what can I do [21:13:55] oh [21:14:04] Amir1: Elastica extension is the source https://gerrit.wikimedia.org/r/#/c/339322/ [21:14:07] but the branch is es5 [21:16:25] Amir1: and the change above got merged in master branch https://gerrit.wikimedia.org/r/#/c/342003/ [21:16:36] so Elastica now depends on a composer dependency that does not work on php5.5 [21:16:43] 06Release-Engineering-Team (Long-Lived-Branches), 10ReleaseTaggerBot: Decide how ReleaseTaggerBot fits into the brave new world of long-lived-branches - https://phabricator.wikimedia.org/T141278#3089345 (10demon) 05Open>03declined Considering we declined T89945, I think this can get the same treatment. [21:17:21] fun :D [21:17:32] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089347 (10Paladox) [21:17:53] and the merge commit got tested with mediawiki/vendor [21:17:58] not the composer update [21:18:00] bah [21:18:08] what a mess [21:19:56] legoktm: Oh, related task I've been meaning to do: T137564 [21:19:57] T137564: Include release extensions/skins as submodules of core (maybe vendor too?) - https://phabricator.wikimedia.org/T137564 [21:20:05] (which would greatly simplify the logic in make-release) [21:20:15] hashar: Do you want to revert it? [21:24:05] hashar: hmm, i can adjust the Elastica extension to allow >= 5.5.0, < 5.6, or >= 5.6, but not both [21:24:24] well, maybe both lemme try [21:24:30] vendor repo is using the 5.5.0 version [21:26:34] Hmm, the mysql connector is creating the file as root [21:26:45] i mean root is and not changing it to gerrit2 [21:26:46] lrwxrwxrwx 1 root root 40 Mar 9 20:46 mysql-connector-java.jar -> /usr/share/java/mysql-connector-java.jar [21:27:38] 10Deployment-Systems, 10Scap (Scap3-MediaWiki-MVP), 07Technical-Debt: `scap clean` should delete /var/lib/l10nupdate/caches/cache-$wmgVersionNumber - https://phabricator.wikimedia.org/T119747#3089384 (10demon) [21:47:26] 10Deployment-Systems, 07Wikimedia-log-errors: MWMultiVersion can fatal for bad Host headers - https://phabricator.wikimedia.org/T71419#3089505 (10demon) 05Open>03Resolved a:03demon I fixed this a month ago [22:11:32] 10Scap: DEPLOY_HEAD should be a symbolic ref - https://phabricator.wikimedia.org/T146062#3089593 (10demon) This isn't really possible right now because we store a ton of extra info in `DEPLOY_HEAD` that's not just a name of a reference. Furthermore, it *must* be a ref, it cannot be a sha1. From the manpage itsel... [22:22:31] 06Release-Engineering-Team, 10Wikimedia-General-or-Unknown, 10Wikimedia-Site-requests: Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640#3089629 (10Krinkle) [22:22:44] 10Deployment-Systems, 06Release-Engineering-Team, 10Wikimedia-General-or-Unknown: Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640#1273402 (10Krinkle) [22:24:39] 10Deployment-Systems, 06Release-Engineering-Team, 10Wikimedia-General-or-Unknown: Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640#3089636 (10Krinkle) >>! @ori wrote: >The mapping of image files in the config repository to the Commons image...