[01:54:27] 10Continuous-Integration-Infrastructure, 07Tracking: Jenkins: migrate all jobs from master to slaves (tracking) - https://phabricator.wikimedia.org/T52228#2460435 (10Quiddity) [02:06:20] 10Continuous-Integration-Infrastructure, 07Tracking: Jenkins: Setup Vagrant for some jobs (tracking) - https://phabricator.wikimedia.org/T50407#2460446 (10Quiddity) [02:07:03] 10Continuous-Integration-Infrastructure, 07Tracking: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#2460448 (10Quiddity) [02:42:09] 10Beta-Cluster-Infrastructure, 07Tracking: [beta project] Set up search (tracking) - https://phabricator.wikimedia.org/T36250#2460497 (10Quiddity) [02:53:11] 10Continuous-Integration-Infrastructure, 07Tracking: Jenkins: Set up linting for php, js, py etc. (tracking) - https://phabricator.wikimedia.org/T37585#2460513 (10Quiddity) [03:19:45] 07Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 07Tracking: Migrate current selenium tests to cucumber - https://phabricator.wikimedia.org/T55849#2460588 (10Quiddity) [03:35:31] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 07Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#2460643 (10Quiddity) [03:42:36] 10Beta-Cluster-Infrastructure, 07Tracking: deployment-prep root tracking bug (tracking) - https://phabricator.wikimedia.org/T39079#2460665 (10Quiddity) [03:42:39] 10Beta-Cluster-Infrastructure, 06Operations: [OPS] exim config points to mchenry.wmflabs.org - https://phabricator.wikimedia.org/T38996#2460667 (10Quiddity) [03:42:41] 10Beta-Cluster-Infrastructure, 07Tracking: deployment-prep root tracking bug (tracking) - https://phabricator.wikimedia.org/T39079#424260 (10Quiddity) [03:42:43] 10Beta-Cluster-Infrastructure, 07Tracking: admin tools on beta (tracking) - https://phabricator.wikimedia.org/T39081#2460669 (10Quiddity) [04:16:49] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2460767 (10Quiddity) [04:18:25] Yippee, build fixed! [04:18:25] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #73: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/73/ [05:13:22] 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016): Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2460896 (10mmodell) Here's my draft of a post about the sponsored improvements that have landed in recent weeks: {P3436} [05:32:46] Project mediawiki-core-code-coverage build #2134: 04STILL FAILING in 2 hr 32 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2134/ [06:08:56] 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2460917 (10greg) >>! In T38891#2459000, @greg wrote: > ```lang=irc > 18:03 < legoktm> '-wmgUsePoolCounter' => [ > 18:03 < legoktm> 'default' => false, // T38891 > 18:03 < legoktm> ], > > ```... [06:10:11] 10Beta-Cluster-Infrastructure, 06Labs, 10MediaWiki-General-or-Unknown, 06Operations: Create a poolcounter instance in deployment-prep - https://phabricator.wikimedia.org/T112501#2460920 (10greg) [06:10:13] 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2460919 (10greg) [06:21:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:43:29] 10Beta-Cluster-Infrastructure, 06Operations: [OPS] exim config points to mchenry.wmflabs.org - https://phabricator.wikimedia.org/T38996#2460945 (10hashar) [06:43:31] 10Beta-Cluster-Infrastructure, 07Tracking: deployment-prep root tracking bug (tracking) - https://phabricator.wikimedia.org/T39079#2460944 (10hashar) [08:29:33] !log deploying 0e9555f to ores-beta (sca03) [08:29:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:37:00] redis in beta is down [08:37:12] deployment-ores-redis [09:05:09] !log rebooting deployment-ores-redis [09:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:09:32] and ores-beta is back up! [09:09:49] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#1711974 (10fgiunchedi) wohooo that's awesome @AlexMonk-WMF ! thanks for working on this and completing it! [09:23:49] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2461366 (10Mglaser) Unit tests fail in REL1_23 branch. See https://gerrit.wikimedia.org/r/#/c/298925/ as an example. It seems that phpunit cannot be accessed:... [11:28:49] Yippee, build fixed! [11:28:50] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #81: 09FIXED in 56 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/81/ [11:33:10] (03PS3) 10Zfilipin: WIP Fix selenium-Wikibase Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/291907 (https://phabricator.wikimedia.org/T128190) [11:33:19] (03PS4) 10Zfilipin: WIP Fix selenium-Wikibase Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/291907 (https://phabricator.wikimedia.org/T140096) [11:42:49] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin: selenium-Wikibase Jenkins job fails with `no such file to load -- features/support/pages (LoadError)` - https://phabricator.wikimedia.org/T140096#2461688 (10zeljkofilipin) My comment on my commit in Gerrit ([[ https://ger... [12:12:53] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 07WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2461718 (10hashar) [12:12:56] 10Beta-Cluster-Infrastructure, 10DBA, 13Patch-For-Review, 07WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2461716 (10hashar) 05Open>03Resolved Confirmed that both db1 and db2 MySQL send syslog. Messages ends up... [12:17:53] hasharAway i was thinking that openstack may have resolved the zuul issue by upgrading gerrit since they found the problem around the same time in december may a bit before upgrading gerrit but since they upgraded gerrit it seems to have fixed it for them, maybe we can just tryed using version 1.4.1 of pycrypto and see if that works for us [12:20:23] but even then we can also try setting paramiko to just 1.8.0 and try that [12:22:53] Or maybe try paramiko 2.0 [12:22:58] please [12:31:24] Or install these libarys GMP and MPIR please [13:02:18] (03PS1) 10Paladox: 2.1.0-151-g30a433b-wmf3precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) [13:02:31] hasharAway ^^ [13:14:58] hashar according to https://github.com/paramiko/paramiko/blob/1.17/setup.py#L79 it misses the 2.4 version so the above patch may fix it hopefully. [13:21:05] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2456573 (10Danny_B) [[ https://phabricator.wikimedia.org/maniphest/qu... [13:25:23] Hello people, I am migrating the analytics refinery repo to scap3 and I have some questions about authentication between tin and the target host [13:26:07] (I am reading https://wikitech.wikimedia.org/wiki/Scap3/Migration_Guide) [13:27:14] for example, my repo will live on tin and it will be deployed to analytics1027, that doesn't have any deploy-service user.. even if I use one already in the system, how should I manage the ssh keys in the key holder (not sure if called in this way)? [13:32:48] hasharAway https://github.com/dlitz/pycrypto/commit/19dcf7b15d61b7dc1a125a367151de40df6ef175#diff-c451844a86998caa3eb2bac332b7d384 [13:36:50] jdlrobson: I remember seeing link to "how to debug jenkins jobs" page you have created recently, but I can not find it now :( [13:37:08] I want to update a thing or two [13:38:15] found it! (I think) https://www.mediawiki.org/wiki/Reading/Web/QA#Simulating_browser_test_run_on_an_unmerged_patchset. [13:42:30] Im going to build the precise dpkg for zuul now on jenkins-slave-01 [13:50:59] (03PS2) 10Paladox: 2.1.0-151-g30a433b-wmf3precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) [13:55:49] hasharAway ive built a dpkg for precise using the above [13:56:18] It's now ready in /var/cache/pbuilder/result/precise-amd64 on jenkins-slave-01 [13:56:20] :) [13:57:05] zuul_2.1.0-151-g30a433b-wmf3precise1_amd64.deb [13:57:13] would you be able to try it please [14:07:14] paladox: it is holiday for hasharAudio [14:07:25] so he is probably having fun with his family :) [14:07:29] Oh thanks for explaning [14:07:40] :) [14:07:45] I made the same mistake in the morning :) [14:07:50] Oh [14:08:13] Ive been looking into the zuul issue, not sure what causes it but may have some reasons why. [14:08:25] So i built a dpkg of https://gerrit.wikimedia.org/r/298957 [14:08:32] hoping that will resolve the issue. [14:08:42] sorry, I am into something completely different, I know little about zuul [14:09:08] ok [14:35:07] 07Browser-Tests, 10Wikidata, 15User-zeljkofilipin: selenium-Wikidata fails with Timeout::Error - https://phabricator.wikimedia.org/T140098#2462138 (10zeljkofilipin) a:05zeljkofilipin>03None [14:36:03] 07Browser-Tests, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin: selenium-Wikidata Jenkins job fails with MediawikiSelenium::ConfigurationError - https://phabricator.wikimedia.org/T140097#2462140 (10zeljkofilipin) a:05zeljkofilipin>03None [14:36:15] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin: selenium-Wikibase Jenkins job fails with `no such file to load -- features/support/pages (LoadError)` - https://phabricator.wikimedia.org/T140096#2462141 (10zeljkofilipin) a:05zeljkofilipin>03None [14:47:23] (03CR) 10Hashar: [C: 04-1] "Precise already has python-crypto 2.4.1 + Ubuntu patches. I fail to see how that would resolve T137525 :(" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) (owner: 10Paladox) [14:47:57] paladox: that is not going to do it [14:48:08] hashar oh [14:48:26] hashar maybe upgrading then [14:48:46] But i thought it was built using 2.6.1 looking at the files in diffusion [14:48:48] yeah maybe [14:48:49] or not [14:49:11] there is no point in trying upgrading packages / cherry picking random patches without understanding what is the issue at hand [14:49:19] Oh yep [14:49:37] But i doint know why zuul is not telling us why it stalled and stopped working [14:50:05] the most likly solution is the new zuul version does not like gerrit 2.8 since looking at upstream openstack upgraded to gerrit 2.11 [14:50:15] around the same time as the connections updates [14:50:43] hashar what about this https://github.com/dlitz/pycrypto/commit/19dcf7b15d61b7dc1a125a367151de40df6ef175#diff-c451844a86998caa3eb2bac332b7d384 [14:51:06] Looks like it edits the files that are in the thread . [14:54:35] that is unrelated [14:54:47] zuul-server worked fine running in the foreground [14:55:01] but as a background daemon, that triggers some kind of deadlock in paramiko / random generator [14:55:05] that is my best estimate [14:55:20] Oh ok [14:55:37] https://phabricator.wikimedia.org/P3423 [14:55:44] Are we using a different paramiko version in the zuul version that didnt worked correctly [14:55:51] compared to the one were currently using [14:55:52] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, and 3 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2462267 (10elukey) @thcipriani thanks! I have another doubt: we'd want to use the deploy-analytics user to perform the deployment but I am not sure... [14:55:53] the first thread is zuul establishing the connection to Gerrit over SSH [14:55:58] Yep [14:56:05] the random generator thread is waiting for the lock to release (line 21) [14:56:15] yep [14:56:41] and the 2nd thread I hvae no idea what it is , but that must be paramiko spawning a thread to handle the ocnnection [14:56:58] Oh [14:57:07] and it is blocked trying to read(32*32) (at line 41) apparently from /dev/urandom [14:57:21] and while that 2 nd thread is blocked [14:57:28] Oh [14:57:29] the 1st one cant process further since it is locked [14:57:33] Oh [14:57:52] the ten million dollars question is why the read on /dev/urandom fails :) [14:58:12] the 3rd thread (lines 43+) is merely for information [14:58:29] that is an handler processing a Queue holding events received from Gerrit [14:59:40] Oh [15:02:53] hashar maybe trying to 2.0 https://github.com/openstack-infra/zuul/commit/30a433b202a65dd7ddcc6eebf4d34041f6743740 [15:02:58] paramiko [15:03:18] paladox: forget about it please. [15:03:23] Ok [15:03:26] that is totally unrelated [15:03:40] ok [15:03:42] that patch is for when folks install zuul using pip [15:03:49] which download whatever version from pypi [15:03:54] oh ok [15:03:56] so if you have paramiko>=1.7 [15:04:00] and that pypi has 2.0 [15:04:03] pip will install 2.0 [15:04:06] Yep [15:04:22] but Zuul code is/was not compatible with paramiko 2.0 so the commit restrict it to <2.0 [15:04:28] ok [15:04:30] and sorry [15:04:36] our Zuul package uses the Precise python-paramiko deb package :) [15:04:46] oh [15:04:59] that's why it probaly works in jessie [15:05:20] since it wont use python-paramik unless im wrong [15:10:39] hashar sorry about that. [15:11:54] ostriches hi, i use ios 10 safari kept taking me to www.gerrit-new.wikimedia.org [15:12:00] which i doint know why [15:12:09] but i removed the www. manually [15:12:24] Meh. [15:13:13] (03Abandoned) 10Paladox: 2.1.0-151-g30a433b-wmf3precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) (owner: 10Paladox) [15:14:05] ostriches could you run puppet compiler on https://gerrit.wikimedia.org/r/#/c/298710/ please [15:14:06] :) [15:14:26] Are you able to run it? https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build [15:14:31] It's a very useful tool if you can! [15:15:02] Running now though https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/3336/ [15:15:49] ostriches i carn't run it [15:15:55] Oh bummer, p [15:15:57] *ok [15:16:00] I was just wondering [15:16:01] Ok [15:16:09] Results! [15:16:10] https://puppet-compiler.wmflabs.org/3336/ [15:16:14] Yep [15:16:16] and thankyou [15:16:16] :) [15:23:59] ostriches i made the same syntax mistake as you [15:24:02] i forgot to do :: [15:24:27] :) [15:27:13] It happens lol. [15:27:19] Yep [15:28:04] ostriches you may want to go with gerrit 2.12.3 which fixes a security issue with mail [15:28:17] and also fixes other things that im not sure if anyone would notice [15:28:35] Thankyou for reviewing one of the patches [15:28:36] :) [15:29:14] 2.12.3 will be super easy after we're done and will probably do it basically immediately after. [15:29:23] I'm just sticking with 2.12.2 for now since we already have it packaged and my momentum is there [15:29:53] Ok [15:29:55] :) [15:30:30] I hope we get the in web editing in differential. [15:31:53] ostriches when you have time would you be able to rerun puppet compiler on https://gerrit.wikimedia.org/r/#/c/298710/ please. [15:31:58] I fixed the syntax problem now [15:32:21] so there should be one config change on the old gerrit server instead of 3 and 3 config changes on lead. [15:32:22] :) [15:42:27] 10Continuous-Integration-Infrastructure: npm jobs failing with ">> Error: Cannot find module 'cst'" in VE-MW and OOjs - https://phabricator.wikimedia.org/T140367#2462408 (10Jdforrester-WMF) [15:42:46] 10Continuous-Integration-Infrastructure, 10OOjs, 10VisualEditor: npm jobs failing with ">> Error: Cannot find module 'cst'" in VE-MW and OOjs - https://phabricator.wikimedia.org/T140367#2462420 (10Jdforrester-WMF) p:05Triage>03High [15:47:13] 10Continuous-Integration-Infrastructure, 10OOjs, 10VisualEditor: npm jobs failing with ">> Error: Cannot find module 'cst'" in VE-MW and OOjs - https://phabricator.wikimedia.org/T140367#2462426 (10Jdforrester-WMF) Looks like this is an issues for grunt-jscs 3+; it was fixed in jscs 3.0.6 but a version of gru... [15:47:38] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #79: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/79/ [15:49:17] hashar: Do you know, why https://gerrit.wikimedia.org/r/#/c/251312/ is not testing actually? [15:50:20] Luke081515: https://integration.wikimedia.org/zuul/ looks fine at least [15:50:40] * hashar heads in debug log for change 251312 [15:51:01] hashar: yep, but I looked at that all the time, and even if the queue is emtpy, no test of that started [15:51:10] Exception: Dependency cycle detected: 251312 in [u'251312', u'287141'] [15:51:15] heh :) [15:51:18] Luke081515: sorry that one is not reported back [15:51:19] o.O [15:51:25] there is some kind of cycle apparently [15:51:41] with Wikibase change https://gerrit.wikimedia.org/r/#/c/287141/ [15:51:50] so imho [15:51:50] hm [15:51:52] 10Continuous-Integration-Infrastructure, 10OOjs, 10VisualEditor, 13Patch-For-Review: npm jobs failing with ">> Error: Cannot find module 'cst'" in VE-MW, OOjs, and OOUI - https://phabricator.wikimedia.org/T140367#2462439 (10Jdforrester-WMF) [15:51:57] the mediawiki core patch is supposed to land first [15:52:07] and thus you could drop Depends-On: I7eaee0170a8d4938c94d1024f9e6f58c94db9c3d [15:52:15] ok, will do [15:52:32] it is merely a way for Zuul to protect itself [15:52:54] and there is no way one can find out about the cycle without looking at the log or at each of the depends-on changes :( [15:52:58] hashar i thought it is a bug [15:53:11] hashar: ah, queued now, thank you [15:53:16] paladox: there is another one related to cycles [15:53:22] Oh [15:53:23] Luke081515: you are magic!! :] [15:53:34] hashar: is there maybe a way to search for "cycle detected" at the logs, and then post information at a change? [15:53:44] maybe a good workaraound [15:53:48] *workaround [15:54:54] for example if we would run a cronjob, and post results at the specific change, or here at IRC, we can fix that faster ;) [15:56:00] here is an exception raised [15:56:04] but I guess it is discarded [15:56:06] * Luke081515 is a bot-owner, so he mostly thinks about a soluation with bots ;) [15:56:17] *solutions [15:58:35] Luke081515: looks like https://phabricator.wikimedia.org/P3442 [16:00:30] so if we search for Exception: Dependency cycle detected: we can find all these patches [16:00:45] but I don't know how much errors like this we have. Do you know that? [16:00:47] but that is in the Zuul server side logs [16:01:05] so the idea is when a Gerrit event is received, the patchset is analyzed [16:01:15] and if that causes a dependency cycle the events is dropped entirely [16:01:24] the main loop does not have even see it so cant really report [16:01:53] hm :-/. How often people report a problem like this? [16:02:12] I mean if it is not often, there is no real need for such a script ;) [16:02:14] that is the 2nd time I hear about a report :) [16:02:20] I am doing the grep in the zuul logs [16:02:39] since June 14th or a month ago: 1 occurence on June 16th [16:03:03] 7 occurences in the current log [16:03:03] ok, so no real need ;) [16:03:10] yeah I guess :=] [16:03:18] makes that easy :) [16:03:45] eventually one will figure out a way to send logs of python softwares to logstash [16:04:56] https://pypi.python.org/pypi/python-logstash that one makes it so UDP packets are sent to logstash [16:05:08] https://pypi.python.org/pypi/logstash_formatter format the packet has a json blob suitable for logstash [16:05:09] magic [16:05:58] I am off again [16:09:31] (03CR) 10Chad: "Per IRC with Tyler: This is probably fine, just need to update the train documentation." [tools/release] - 10https://gerrit.wikimedia.org/r/298784 (owner: 10Chad) [16:11:57] ostriches ive also updated https://gerrit.wikimedia.org/r/#/c/298979/ [16:11:58] :) [16:17:14] (03Restored) 10Paladox: 2.1.0-151-g30a433b-wmf3precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) (owner: 10Paladox) [16:17:28] (03PS3) 10Paladox: 2.1.0-151-g30a433b-wmf3precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) [16:17:48] (03CR) 10Paladox: "@hashar ^^" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/298957 (https://phabricator.wikimedia.org/T137525) (owner: 10Paladox) [16:44:45] hasharAway sorry that im on continuing to do zuul, but one last thing im trying to install version 2.6.1 instead of using the deb since on jessie which uses that version it seems to works but im getting this error [16:44:45] src/MD2.c:31:20: fatal error: Python.h: No such file or directory [16:44:46] compilation terminated. [16:44:46] error: command 'gcc' failed with exit status 1 [16:44:56] warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath. [16:44:56] building 'Crypto.Hash._MD2' extension [16:45:32] ---------------------------------------- [16:45:32] Can't roll back pycrypto; was not uninstalled [16:45:32] Cleaning up... [17:05:24] paladox: Hey! What is your question? [17:06:03] If you try to compile software and a requirement isn't available (like Python.h), install the package that provides that requirement (includes that file). [17:06:34] andre__ I was wondering if hashar knew the problem since im trying to trace why zuul zuul_2.1.0-151-g30a433b-wmf1precise1_amd64.deb wont work for us [17:06:50] andre__ yep i searched online and it said python-deb [17:06:54] *dev [17:06:56] paladox, knowing "which problem"? [17:07:01] ah. So... install it? [17:07:03] Why it wont compile [17:07:06] It's a generic question. [17:07:06] I have [17:07:09] and same problem [17:07:23] paladox, so what package did you install (exact name), and how (exact command)? [17:07:36] and on which OS / distro are you? [17:07:41] Well i did apt-get install python-deb and im on jessie [17:07:47] compiling it for precise [17:08:14] I assume that "jessie" is a Debian version. [17:08:39] Yes [17:08:54] paladox: what is the output of: dpkg -l python-dev [17:09:11] dpkg -l python-dev [17:09:15] Desired=Unknown/Install/Remove/Purge/Hold [17:09:15] | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend [17:09:15] |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) [17:09:15] ||/ Name Version Architecture Description [17:09:15] +++-==============-============-============-================================= [17:09:15] ii python-dev 2.7.9-1 amd64 header files and a static library [17:11:14] paladox: so where is Python.h located / is it actually on your disk? Did you check? [17:11:33] I haven't checked if it is, how do i check please [17:11:39] if you have "locate" installed and its database is up-to-date: locate Python.h [17:11:50] or: find / -name 'Python.h' 2>/dev/null [17:11:54] which can take a while. [17:12:13] Thanks [17:12:34] find / -name 'Python.h' 2>/dev/null [17:12:43] /usr/include/python2.7/Python.h [17:13:02] /usr/include/python3.4m/Python.h [17:26:20] 05Gitblit-Deprecate, 06Release-Engineering-Team, 10Diffusion, 07WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2462948 (10greg) [17:26:23] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2462946 (10greg) 05Resolved>03Open (Reopening while we discuss this.) >>! In T89940#2451956, @Paladox wrote: > @bd808 we could stop those commits from being linked in tasks through an update... [17:30:49] 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016): Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2463000 (10Aklapper) @mmodell: Thanks for working on this. Very appreciated! Some quick'n'dirty blunt feedback: * s/Phacility/Phacility (th... [17:34:59] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2463071 (10demon) >>! In T89940#2462946, @greg wrote: >> On IRC @Paladox made a claim that the external viewer links are valuable for reviewing code from phones and other unspecified user-agents t... [17:35:33] paladox: Which exact command did you use for compiling? [17:35:51] 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016): Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2463076 (10greg) I asked @mmodell last night if I could make an etherpad to do some light copy/editing. I go distracted (it was almost 11pm)... [17:35:55] andre__ i ran DEB_BUILD_OPTIONS=nocheck GIT_PBUILDER_AUTOCONF=no DIST=precise WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder [17:36:26] Hmm, and there my knowledge ends I'm afraid. :-/ [17:36:30] Ok [17:36:50] Wondering if having both Python2 and Python3 dev packages installed might end up picking the wrong one. But really just guessing [17:37:26] Ok [17:37:30] thanks for your time [17:37:43] Sorry, I only compile on rpm instead of deb systems and classic "configure/make/make install" stuff hence no idea about "git-buildpackage" ... [17:37:49] Oh [17:37:51] Heh, sure sure, you are very welcome! :) [17:37:55] :) [17:38:41] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2463083 (10mmodell) The one use case I have for `refs/changes` in phabricator is for converting gerrit unmerged changes into differential diffs. [17:40:54] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2463085 (10demon) >>! In T89940#2463083, @mmodell wrote: > The one use case I have for `refs/changes` in phabricator is for converting gerrit unmerged changes into differential diffs. There's a d... [17:48:07] Project mediawiki-core-code-coverage build #2135: 04STILL FAILING in 2 hr 48 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2135/ [17:53:53] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2463162 (10mmodell) >>! In T89940#2463085, @demon wrote: >>>! In T89940#2463083, @mmodell wrote: >> The one use case I have for `refs/changes` in phabricator is for converting gerrit unmerged chan... [18:47:32] how might i query hiera on beta cluster? Some of the elasticsearch servers arn't being configured correctly and i suspect the appropriate vars from hiera arn't being set, but trying `hiera -c /etc/puppet/hiera.yaml elasticsearch::publish_host` only gets me two errors about cannot load backend mwyaml and yunaml [18:47:49] (also tried some other keys to double check, like mediawiki::web::sites::domain_suffix but same) [18:54:20] !log deployment-prep manually edited elasticsearch.yml on deployment-elastic05 and restarted to get it listening on eth0. Still looking into why puppet wrote out wrong config file [18:54:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:14:38] Seems Jenkins/phab isn't working properly. https://phabricator.wikimedia.org/D290 - https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/18/console. Arc config error. [19:14:44] If it isn't meant to work yet, maybe it should be disabled. [19:54:01] Krinkle: there is a problem with .arcconfig that I've been trying to solve. https://gerrit.wikimedia.org/r/#/c/298097/ was supposed to fix it but apparently not [19:57:14] actually this is a different problem [20:05:36] Krinkle: it IS meant to work, and I'm working on it [20:08:48] twentyafterfour: okay, cool. :) I didn't know it was supposed to work yet, thoguht maybe it was an one-off experiment that was left enabled. [20:09:30] ostriches yep i fixed the syntax error now, http://puppet-compiler.wmflabs.org/3339/ytterbium.wikimedia.org/ [20:09:42] shows that instead of 4 configs changes for the old gerrit server [20:09:45] it is now 2 [20:09:45] twentyafterfour: btw, it seems https://github.com/wikimedia/oojs-router is behind https://phabricator.wikimedia.org/diffusion/GOJR/ - how long should replication take? We landed 2 commits in the past hour. [20:10:03] Krinkle: https://gerrit.wikimedia.org/r/#/c/299029/ should fix the problem you linked to [20:10:08] I'm excited to see Jenkins testing for Diffusion. Cool stuff :) [20:10:31] Krinkle: replication shouldn't take that long... [20:11:13] phabricator says it's up to date, 15 seconds of lag according to https://phabricator.wikimedia.org/diffusion/GOJR/manage/status/ [20:11:38] but replication is in gerrit not phabricator [20:11:43] (github replication) [20:11:52] and I have no clue about gerrit replication [20:12:10] I can move the repo to diffusion and disable gerrit's replication [20:12:34] Differential* (I know I know) [20:12:50] (that ^ was about 3 minutes late, oops, in a meeting) [20:12:53] greg-g: I meant diffusion [20:13:05] Krinle meant DIfferential ;) [20:13:13] ah [20:13:15] yeah [20:13:28] actually it's both - diffusion and differential both integrate with harbormaster [20:13:37] Phabricator* [20:13:38] :) [20:13:41] yeah [20:14:32] stuff integrates with other stufff [20:14:32] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#2463873 (10demon) One liner for all extensions and skins meta repos: git submodule foreach "git branch -r | grep -Eo 'wmf/.*' | gre... [20:14:34] for an example that isn't broken, see https://phabricator.wikimedia.org/diffusion/PHEX/history/wmf%252Fstable/ [20:19:37] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#22541 (10mmodell) I agree, all those old tags and branches aren't very useful after a while. [20:25:37] Yay i got graphite and grafana running on labs [20:25:39] http://gerrit-grap.wmflabs.org/login [20:25:41] http://gerrit-graph.wmflabs.org/login [20:28:04] paladox_: cool [20:28:50] :) [20:32:08] Krinkle: weird, I can't figure out how jenkins has a home directory of '/home/jenkins' on the slaves. The user is supposed to be 'jenkins-slave' anyway... [20:32:25] Maybe arc is looking in the wrong place? [20:32:52] Krinkle: maybe, but I did 'echo $HOME' in a jenkins job and ran it, just to figure out where it's supposed to be [20:34:41] Also there's this at the top of the console output: "Building remotely on ci-jessie-wikimedia-182791 (ci-jessie-wikimedia) in workspace /home/jenkins/workspace/phabricator-jessie-diffs" [20:34:54] but operations/puppet has zero references to '/home/jenkins' [20:35:04] so it's somehow done outside of puppet :( [20:43:00] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #85: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/85/ [20:43:02] twentyafterfour: Hm.. does nodepool actually use the slave puppet class? [20:43:09] I thought it had its own image builder and all [20:43:15] and different roles/classes [20:43:20] (some overlap, but not all) [20:43:28] well yes exactly, but I couldn't find anything in the imagebuilder stuff either [20:43:36] twentyafterfour: If that is the case, is it possible that Puppet creates the user with default settings? [20:43:56] Krinkle: that may be exactly what's happening [20:43:57] Afaik it needs to be declared something with 'user{}' in order to require User[] it. [20:44:05] But maybe it's different for users. [20:44:41] yeah, I don't know what's going on exactly, and I haven't been able to catch Hashar for long enough to figure it out [20:48:08] assign him a task :) [20:54:24] greg-g: done (https://phabricator.wikimedia.org/T140417) [20:55:13] 10Continuous-Integration-Config, 07Jenkins, 07Puppet: jenkins homedir on nodepool slaves is in /home/jenkins but this doesn't seem to be anywhere in puppet - https://phabricator.wikimedia.org/T140417#2464009 (10mmodell) [20:55:40] 10Continuous-Integration-Config, 07Jenkins, 07Puppet: jenkins homedir on nodepool slaves is in /home/jenkins but this doesn't seem to be anywhere in puppet - https://phabricator.wikimedia.org/T140417#2463982 (10mmodell) [21:00:01] :) [21:34:20] http://gerrit-grap.wmflabs.org/dashboard/db/new-dashboard [21:34:24] twentyafterfour ^^ [21:34:25] :) [21:50:40] 10Continuous-Integration-Config, 07Jenkins, 07Puppet: jenkins homedir on nodepool slaves is in /home/jenkins but this doesn't seem to be anywhere in puppet - https://phabricator.wikimedia.org/T140417#2464260 (10hashar) The user for Jenkins are absolutely a complete mess. That boils down to: | User | Realm... [21:54:50] 10Continuous-Integration-Config, 07Jenkins, 07Puppet: jenkins homedir on nodepool slaves is in /home/jenkins but this doesn't seem to be anywhere in puppet - https://phabricator.wikimedia.org/T140417#2464272 (10mmodell) @hashar: Thanks for the detailed response. I'd like to make this a little less messy but... [23:12:05] 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2464680 (10Mattflaschen-WMF) 05Resolved>03Open Happened again. I worked around it by rebooting in wikitech, but shouldn't keep happening. [23:16:57] 10Beta-Cluster-Infrastructure: deployment-fluorine becomes unresponsive frequently - https://phabricator.wikimedia.org/T140313#2464706 (10greg) p:05Triage>03High a:05AlexMonk-WMF>03None