[00:14:58] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2915304 (10srishakatux) @Rfarrand @Qgil using the documentation from... [00:16:35] RECOVERY - Parsoid on deployment-parsoid09 is OK: HTTP OK: HTTP/1.1 200 OK - 1014 bytes in 0.078 second response time [00:28:56] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2915323 (10srishakatux) [03:06:33] is "ImportError: No module named sh" from scap on tin a known bug? [03:08:58] output at -- https://phabricator.wikimedia.org/P4701 [03:28:25] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:18] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31806 bytes in 1.404 second response time [04:48:05] 10Continuous-Integration-Infrastructure: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#2915815 (10Krinkle) [05:00:45] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other, 07Mobile, 13Patch-For-Review: CommentStreams: The module 'ext.CommentStreams' must not have target 'mobile' because its dependency 'jquery.ui.dialog' does not have it - https://phabricator.wikimedia.org/T151863#2915827 (10Krinkle) 05Open>03R... [05:17:10] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:17:38] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:18:32] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:18:54] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:21:10] PROBLEM - Puppet run on deployment-salt02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:56:10] RECOVERY - Puppet run on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:57:08] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:57:34] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [05:58:32] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:58:53] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:34:53] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916063 (10jcrespo) > we will not be able to use jdbc for this as it is not supported, so we will need to set it server side. Again, this is not a... [09:41:05] !log integration: pruning /srv/pbuilder/aptcache/ on Jessie perm slaves [09:41:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:41:39] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2644059 (10Joe) I might be very unpopular with this opinion, but: I think emojis in code reviews should be avoided. Plain old written text is the w... [10:41:53] hashar: how hard do you think it would be to get phan for extensions going? [11:12:08] time() ? [11:12:29] addshore: I haven't followed the effort to add phan [11:12:43] but seems it work all fine for mw/core [11:12:51] okay, yeh, I might take a stab at it today [11:12:57] doesnt feel like it should be too hard :) [11:13:52] addshore: the trick though is the phan wrapper is in mediawiki/core [11:14:05] but is that not the same for phpunit? [11:14:13] yeah [11:14:22] so gotta craft a job that clone core + the extension or skin [11:14:46] do the composer dance then check whether ./tests/phan/bin/phan accepts extra arg to point to the extension [11:14:52] maybe it is all about doing: [11:14:57] cd $WORKSPACE/src [11:15:25] so yeh, its just ./tests/phan/bin/phan ./extensions/FOO [11:15:25] ./tests/phan/bin/phan -m checkstyle extensions [11:15:54] I think Erik B made the phan wrapper to accept any arg [11:16:05] cool [11:45:11] hashar: just checking (as it has been a while since I have done jjb stuff) $ZUUL_PROJECT in the case of a test for the wikibase extension will be Wikibase ? [11:45:51] addshore: ZUUL_PROJECT is the full Gerrit project name [11:45:52] oh, no, it will be mediawiki/extensions/Wikibase? [11:45:55] okay! [11:45:56] hence mediawiki/extensions/Wikibase [11:46:20] isn't the phan wrapper able to traverse "$WORKSPACE/src/extensions" ? [11:46:25] regardless of the extension name ?e [11:46:32] else, we have zuul inject EXT_NAME on some job [11:46:40] thats not what would be wnated then, as each extension has its own phan config [11:47:10] so you pass the extension dir to the pahn script, which reads the phan config for that extension and runs phan in the way configured for that extension [11:47:50] ohhh [11:48:12] is EXT_NAME generally available? [11:48:21] (checking) [11:48:45] zuul/parameter_functions.py [11:48:59] set_mw_dependencies() [11:49:09] is applied on mediawiki/extensions/* and mediawiki/skins/* [11:49:16] and will set either SKIN_NAME or EXT_NAME [11:49:27] awesome! [11:50:26] (03PS1) 10Addshore: WIP DNM phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 [11:50:28] rough draft ^^ [11:53:56] I actually think that could work.... [11:54:27] (03PS2) 10Addshore: Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) [12:00:13] (03PS1) 10Addshore: Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 [12:00:23] hashar: ^^ let me know your thoughts on testing that out! :) [12:01:48] eeeeek [12:02:12] addshore: is phan going to require the extensions dependencies ? [12:02:19] yup [12:02:23] ;-( [12:02:24] in some cases anyway [12:02:47] for example is an extension needs BetaFreatures, it needs to see the BF class inorder for phan to not complain [12:03:00] got it [12:03:08] also needs all of the composer installed stuff / anything it might use xD [12:03:37] (03CR) 10Hashar: "Lets give it a try :}" [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) (owner: 10Addshore) [12:03:44] addshore: lets do that :} [12:03:49] have you deployed the job? [12:03:53] can be tested with https://gerrit.wikimedia.org/r/#/c/330260/ [12:04:05] * addshore cant deploy CI stuff ;) [12:06:07] ahh wait, pattern: 'src/tests/phan/issues/latest' [12:06:08] thats wrong [12:06:08] bah [12:06:36] should be src/extensions/$EXT_NAME/tests/phan/issues/latest [12:06:53] but EXT_NAME won't work there right? [12:07:24] oh [12:07:27] yeah so that is the annoying part [12:07:34] EXT_NAME is set by Zuul [12:07:41] and thus if you trigger the build manually .. that fails [12:07:47] cause the env variable is not set [12:07:51] I mean, we could move the 'latest' file up a few dirs in the shell builder? [12:08:01] oh... xD [12:08:04] or yeah [12:08:11] you can move the file under $WORKSPACE/log/ [12:08:17] and thus it will also be archived [12:08:23] oooh, cool! [12:11:24] (03PS3) 10Addshore: Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) [12:11:28] right, that looks better [12:11:50] (03PS4) 10Addshore: Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) [12:13:19] and for EXT_NAME not being in manually triggered builds, does that mean manually as in through jenkins or manually via commenting on gerrit? as commenting should go through zuul right? [12:16:54] addshore: manually via Jenkins Rebuild link [12:22:19] hashar: okay, so testing through "check experimental" would still work? [12:33:06] addshore: yeah [12:33:11] that will be processed by Zuul [12:33:14] which runs the parameter function [12:33:20] craft EXT_NAME and inject it in the job [12:34:22] I have wasted half an hour due to a Safari bug not invalidating file:// resources that changed on disk bah :((( [12:34:32] going to grab something to eat [12:34:34] bbl [12:35:41] :D [13:37:53] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916378 (10Aklapper) (Same feelings here as Joe: Task could be declined.) [13:40:08] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916379 (10Paladox) But utf8 is not full unicode. It's better to just fix it then to leave it. Yes we shouldn't bug dba but this is a db thing. [13:48:41] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2916388 (10hashar) This is mostly done. Both deployment-prep and CI have the patch cherry picked. Have to migrate the #puppet-compiler... [13:54:51] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + maybe accounts - https://phabricator.wikimedia.org/T154205#2916409 (10Aklapper) >>! In T154205#2912984, @Paladox wrote: > Apparently the way they fixed it was the wrong... [13:59:03] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + maybe accounts - https://phabricator.wikimedia.org/T154205#2916416 (10Paladox) I did where I said per Luca at https://gerrit-review.googlesource.com/#/c/93479/ [14:10:03] I'm not sure why, but my tox-jessie build is failing after including 'py36' in the tox.ini. Is this the right place to get help? https://integration.wikimedia.org/ci/job/tox-jessie/15089/console [14:10:11] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2916432 (10hashar) All CI slaves attached to standalone puppet masters have been migrated. The last one to move is the puppet compiler. [14:11:02] schana: there is no python 3.6 on the CI instances :( [14:11:21] right, but my tox config should ignore non-present interpreters [14:11:37] https://gerrit.wikimedia.org/r/#/c/330408/1/tox.ini [14:11:55] ah yeah [14:12:20] recommendation/api/types/test/conftest.py:1: ImportWarning: Not importing directory '/home/jenkins/workspace/tox-jessie/recommendation/test': missing __init__.py [14:12:23] from recommendation.test.conftest import change_config_and_setup_responses, config_locations # NOQA [14:12:42] hmm [14:12:59] maybe the tox version we use doesn't support py36 [14:13:45] schana: you can verify by adding in the commands = python --version [14:17:05] okay hashar, I'll do that and see what happens [14:17:17] schana: trying locally as well [14:17:47] schana: yeah it uses python 2.7 [14:17:56] ============================= test session starts ============================== [14:17:56] 13:44:41 platform linux2 -- Python 2.7.9, pytest-3.0.5, py-1.4.32, pluggy-0.4.0 -- /home/jenkins/workspace/tox-jessie/.tox/py36/bin/python [14:18:14] weird [14:19:17] schana: a hack would be to set the python to use explicitly [14:19:19] example is https://github.com/WoLpH/mastering-python-code/blob/master/tox.ini#L6-L9 [14:19:30] "basepython" in tox is the executable name it is going to use [14:21:09] schana: for info the CI slaves have tox 1.9.2 [14:21:22] and http://tox.readthedocs.io/en/latest/changelog.html states that tox 2.5.0 added py36 / py37 [14:21:32] ah, okay [14:22:51] https://integration.wikimedia.org/ci/job/tox-jessie/15090/console [14:22:52] success [14:22:56] thanks hashar [14:23:25] \O/ [14:23:55] if only Debian support multiple version of python/php etc [14:46:50] (03PS1) 10Hashar: Switch ops/puppet doc to rake doc [integration/config] - 10https://gerrit.wikimedia.org/r/330415 (https://phabricator.wikimedia.org/T143233) [14:55:39] hashar: so for this phan thing can we just merge and deploy https://gerrit.wikimedia.org/r/#/c/330396/ and https://gerrit.wikimedia.org/r/#/c/330399 and then do some testing with experimental? [14:55:50] yeah [14:55:55] that is what we do usually [14:56:15] (03PS5) 10Hashar: Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) (owner: 10Addshore) [14:56:23] (03CR) 10Hashar: [C: 032] Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) (owner: 10Addshore) [14:56:50] addshore: :-}}} [14:57:03] http://s2.quickmeme.com/img/d9/d9ac09c5da5e04af0bc1aa3cdc5c81402a1d63b5af572eb3a90a0b1875bea336.jpg [14:57:13] (03PS2) 10Hashar: Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 (owner: 10Addshore) [14:57:15] (03CR) 10Hashar: [C: 032] Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 (owner: 10Addshore) [14:58:05] let me know when I can "check experimental" [= [14:58:14] ahhh [14:58:16] quickmeme [14:58:20] ask to login with facebook [14:58:23] ... sorry NO! [14:58:31] really? it doesnt ask me! [14:58:52] https://usercontent.irccloud-cdn.com/file/IaCnnfz2/ [14:59:00] a meme site I like is http://knowyourmeme.com/ [14:59:19] (03Merged) 10jenkins-bot: Phan for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/330396 (https://phabricator.wikimedia.org/T153039) (owner: 10Addshore) [14:59:23] addshore: well your link worked. But to create a new meme it asks to login [15:00:06] (03CR) 10jerkins-bot: [V: 04-1] Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 (owner: 10Addshore) [15:01:53] ahh, okay! [15:02:01] oooh, jenkins says no! [15:02:45] gate keeper! [15:02:58] FAILURE: Job mwext-php70-phan-jessie not defined [15:03:16] oh, I forgett i have to deifne it in there too right? [15:04:10] the job fails because the zuul layout reference a job that does not exist in jenkins [15:04:15] gotta deploy it. Doing so [15:04:23] or you made a mistake somewhere ;D [15:04:23] aaaaahhhh, okay! [15:04:46] (03CR) 10Hashar: [C: 032] Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 (owner: 10Addshore) [15:06:53] pass now [15:07:18] (03Merged) 10jenkins-bot: Add mwext phan job to ext-unittest-generic [integration/config] - 10https://gerrit.wikimedia.org/r/330399 (owner: 10Addshore) [15:07:22] [= [15:07:30] check experimental time? ;) [15:07:59] allmos [15:08:00] t [15:08:13] gotta deploy the config file on the server [15:08:16] and reload zuul [15:08:27] (fab deploy_zuul) from root of integration/config repo [15:08:33] but you need shell access to contint1001 [15:08:36] anyway did it [15:08:47] addshore: GO GO GO !!!!!! [15:08:49] okay, checking on https://gerrit.wikimedia.org/r/#/c/328917/5 [15:09:05] https://integration.wikimedia.org/ci/job/mwext-php70-phan-jessie/1/ [15:10:36] * hashar rolls the drums [15:13:03] finished! failure :D [15:13:12] mv: cannot stat ‘./extensions//tests/phan/issues/latest’: No such file or directory [15:13:35] ./tests/phan/bin/phan ./extensions/ -m checkstyle [15:13:45] looks liek the env var didn't get added :D [15:13:48] brb! [15:15:26] :( [15:15:27] 15:11:57 mv: cannot stat ‘./extensions//tests/phan/issues/latest’: No such file or directory [15:15:29] yeah [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:44:36] (03PS1) 10Hashar: Drop link to puppet source [integration/docroot] - 10https://gerrit.wikimedia.org/r/330423 (https://phabricator.wikimedia.org/T143233) [15:54:17] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916586 (10demon) What @paladox said. Yes, this was filed in context of emojis, but really it's about all extended unicode that mysql/maria's "utf8"... [15:57:42] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mediawiki-extensions-qunit-jessie tests are failing - https://phabricator.wikimedia.org/T153597#2916604 (10hashar) p:05Unbreak!>03Normal >>! In T153597#2891708, @Tgr wrote: > (I still don't understand why https:... [16:01:44] hashar: \o/ [16:01:48] \O/ [16:02:02] here my question :) [16:02:16] I've created a new repository and I would like to know a couple of things [16:02:29] 1) do I need to do anything special to make it appear in diffusion and github? [16:02:51] 2) how to enable/add Jenkins on it? It is a python project and has tox configured [16:03:06] for replication of Gerrit git repos to Diffusion I have no idea but ostriches will know. For github we have to manually create the project iirc [16:03:36] for CI configuration, you would setup CI to run simply "tox" [16:03:58] https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Test_your_Python#Editing_Zuul_configuration :D [16:04:01] so in short: [16:04:06] clone integration/config.git [16:04:10] head to zuul/layout.yaml [16:04:17] Same thing in Diffusion as Github [16:04:21] Gotta create them manually [16:04:26] and add some section for your project that adds tox-jessie. I can do that easily [16:04:27] * ostriches mutters some obscenities [16:04:47] volans: what is the repo name? [16:04:47] ostriches: and should I do it or ask you? :) [16:05:04] hashar: operations/software/cumin [16:05:12] ostriches: I wish we had the gerrit repo creation managed via task in phab so we can track addition of diffusion/github/CI etc [16:05:57] github done https://github.com/wikimedia/operations-software-cumin [16:06:15] great! thanks a lot [16:06:32] (03PS1) 10Hashar: [operations/software/cumin] add tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330426 [16:06:34] and CI ^^ [16:06:55] Yeah should probably track in Phab [16:06:56] (03CR) 10Hashar: [C: 032] [operations/software/cumin] add tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330426 (owner: 10Hashar) [16:06:58] * volans just opened layout.yaml :D [16:07:05] Could make a form to simplify filing it too [16:07:05] thanks again then [16:07:15] volans: and you get some info about tox / venv etc on https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Test_your_Python#Editing_Zuul_configuration [16:07:28] and somehow on https://tox.readthedocs.io/en/latest/ as well [16:07:41] but usually once you have the boilerplate tox.ini set, there is not much to change in it [16:08:02] I've used conftool as a starting point, so I hope it will will work out of the box [16:08:21] ostriches: I think I raised that a year or so ago and folks complained that it was not the wiki way (lack of history, high barrier of entry etc) [16:08:36] (03Merged) 10jenkins-bot: [operations/software/cumin] add tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330426 (owner: 10Hashar) [16:08:50] deployed [16:09:13] volans: well if locally "tox" pass, it should pass on CI :D [16:09:48] hashar: so now that I had already a CR before we added the tox-jessie config... a recheck should be enough or I need to close/reopen the CR? [16:10:06] just comment "recheck" [16:10:18] CI detects that and enqueue the change/patchset exactly as if it had just been sent [16:10:41] then after 5-10 seconds it should appear on https://integration.wikimedia.org/zuul/ [16:10:46] hashar: If people complained they were wrong. [16:10:47] Ugh [16:10:56] Lack of history? Lie [16:11:00] High barrier of entry? Lie [16:11:04] Nothing but FUD there! [16:11:06] ostriches: I can't remember the details. But well yeah a form in Phabricator would be idea [16:11:10] [ ] create repo in Gerrit [16:11:15] [ ] add rights in Gerrit [16:11:16] I can't create forms it would appear! [16:11:21] [ ] create github/diffusion projects [16:11:23] [ ] set CI [16:11:24] etc [16:12:12] volans: set() != list https://integration.wikimedia.org/ci/job/tox-jessie/15096/console -:-} [16:12:23] hashar: interesting tox failed, while locally works [16:12:26] checking the versions [16:13:00] hard to know why [16:13:10] "You do not have permission to create these objects." [16:13:14] though if they are true unit tests, maybe that is due to some changed dependencies [16:13:14] I can't, figures. [16:13:21] hashar: never mind, my bad [16:13:25] ostriches: :-D [16:13:38] I just changed one thing before sendig the CR and forgot to run the tests [16:13:45] the last 1 line change :D [16:19:20] (03PS1) 10Hashar: Inject dependencies on mwext-php70-phan-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330430 [16:19:28] addshore: https://gerrit.wikimedia.org/r/330430 Inject dependencies on mwext-php70-phan-jessie [16:19:36] addshore: forgot about that earlier sorry [16:19:40] (03CR) 10Hashar: [C: 032] Inject dependencies on mwext-php70-phan-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330430 (owner: 10Hashar) [16:20:06] addshore: that will cause Zuul to set EXT_NAME as well as the dependencies variables (and thus the job to clone the various deps) [16:20:10] (03CR) 10Addshore: [C: 031] Inject dependencies on mwext-php70-phan-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330430 (owner: 10Hashar) [16:20:13] :D [16:20:20] epic [16:20:36] epic would be something like: [16:20:54] !wmf-ai DO run phan on all mw extensions and report by email and here [16:20:57] (03Merged) 10jenkins-bot: Inject dependencies on mwext-php70-phan-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/330430 (owner: 10Hashar) [16:21:01] :D [16:21:05] hashar: one day ;) [16:21:09] for sure! [16:21:17] I am just hoping that day is far away [16:21:23] so I don't end up being unemployed [16:21:54] though I will retired a bit before 2045 [16:21:58] let me know when I can do check experi again! [16:22:02] so by that time maybe AI will be all over :( [16:22:09] addshore: deployed! [16:22:22] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916718 (10Paladox) Oh, is that mysql binary or is it utf8 binary? [16:23:47] running at https://integration.wikimedia.org/ci/job/mwext-php70-phan-jessie/2/console [16:25:00] 16:24:40 PHP Warning: DOMDocument::loadXML(): Start tag expected, '<' not found in Entity, line: 1 in /home/jenkins/workspace/mwext-php70-phan-jessie/src/tests/phan/bin/postprocess-phan.php on line 69 [16:25:01] (03PS12) 10Paladox: Create operations-puppet-catalog-compiler job in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/325064 (https://phabricator.wikimedia.org/T97513) [16:25:20] hashar hi, im wondering could you review ^^ please? [16:29:46] paladox: whenI get time yeah [16:29:51] you are rebasing too much :D [16:30:34] addshore: bah :( phan should run with some verbose mode to list out what it is currently processing [16:30:34] (maybe) [16:31:41] ok [16:32:38] addshore: or maybe there is some notice/error emitted by the phan run [16:32:59] and the whole stdout is then piped to postprocess-phan.php which chokes because the first line is not an xml start tag or something like that [16:34:35] hmmm [16:35:04] hashar: can you dive into the workspace and pastebin the script output? [16:35:19] sounds like it might be an idea to get the output printed to stdout.... [16:35:31] addshore: the instances are discarded [16:35:40] ahh yes..... [16:36:03] and the latest file is out moved to the logged area, as it doesnt exist as the script died :D [16:36:13] guess the post processor will need some more robust error handling [16:36:23] such as dumping the input XML on error [16:36:34] or maybe we can save the raw output before it is processed [16:36:40] stash it in /log/ so it get attached to the build [16:36:57] I don't even know what it get in input [16:38:02] I'll take another look in a little bit! :) [16:38:21] OH GOT IT [16:38:27] in mw/core bin/phan [16:38:31] we do something like: [16:38:37] $PHAN | postprocess [16:38:42] we can sneak a tee there [16:38:47] to capture the output :} [16:39:20] or split the post processing in a second exec [16:39:23] instead of using pipe [16:39:53] addshore: in short you can hack bin/phan send it in Gerrit then send a patch to an ext that Depends-On on it and hack [16:39:59] (or easier, reproduce locally) [16:40:06] that is all for today [16:41:45] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916768 (10Dzahn) >>! In T145885#2916065, @Joe wrote: > I think emojis in code reviews should be avoided. > I honestly don't see why this should ev... [16:47:58] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916773 (10jcrespo) > this is a db thing To prove this is **not** a DB server issue, but a client code/driver bug: ``` $ mysql -h m2-master review... [16:51:30] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916775 (10demon) >>! In T145885#2916773, @jcrespo wrote: >> this is a db thing > > To prove this is **not** a DB server issue, but a client code/d... [16:55:22] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916800 (10demon) >>! In T145885#2915101, @Paladox wrote: > Apparently jdbc does not support utf8mb4 > > https://www.google.co.uk/#q=fatal:+++cause... [16:55:56] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916801 (10jcrespo) > altering existing tables/data Altering tables and is not a problem, changing global configuration that affects other database... [17:04:55] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916841 (10Paladox) >>! In T145885#2916800, @demon wrote: >>>! In T145885#2915101, @Paladox wrote: >> Apparently jdbc does not support utf8mb4 >> >... [17:07:41] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916844 (10demon) >>! In T145885#2916841, @Paladox wrote: > oh so doing characterEncoding=utf8mb4 works? Because when i tried that it failed to sta... [17:08:32] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916846 (10Paladox) Oh, continuing to use utf-8 works but pasting an emoji still fails with 500 error. [17:18:08] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916878 (10Dzahn) @paladox What was the solution that resulted in it _not_ showing a 500 error but just a ? instead. Afair there was one? That would... [19:08:36] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917289 (10Paladox) @Dzahn i if we do just &connectionCollation=utf8mb4_unicode_ci that should at least fix the 500 errors but emojis won't show. [19:29:14] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917389 (10Paladox) @Dzahn and @demon setting ?useUnicode=true should stop the 500 errors but emojis won't show for now (db conversion required too) :) [19:30:17] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917395 (10Paladox) https://bugs.mysql.com/bug.php?id=57694 [20:00:11] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T153351#2917523 (10thcipriani) [20:01:08] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T153351#2877832 (10thcipriani) Removed blocking task as it has been partially reverted for the purposes of unblocking this train. Discussion is ongoing on {T153761}. [20:09:46] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:10:46] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:12:05] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:12:25] PROBLEM - Puppet run on deployment-mediawiki06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:12:49] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:12:53] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:13:05] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:13:49] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:13:55] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:13:57] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:14:42] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:15:26] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:16:53] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:17:18] wikitech had a problem and recovered [20:17:25] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:17:29] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:18:30] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:20:35] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:37:47] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:27] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:55] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:57] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:47] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:44] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:04] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:22] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:24] RECOVERY - Puppet run on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:28] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:52] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:53:04] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [20:53:52] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [20:54:42] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:28] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:36] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [20:56:51] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:47] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:59:27] 03Scap3: Add DEPLOY_DIR env var to scap command checks - https://phabricator.wikimedia.org/T154612#2917726 (10thcipriani) [21:05:14] 03Scap3: Add DEPLOY_DIR env var to scap command checks - https://phabricator.wikimedia.org/T154612#2917741 (10dduvall) Should we just make it the cwd during check execution? [21:08:21] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: Connection refused [21:10:47] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:12:52] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917759 (10Aklapper) >>! In T145885#2916586, @demon wrote: > this was filed in context of emojis, but really it's about all extended unicode that my... [21:13:21] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 44323 bytes in 1.328 second response time [21:13:23] PROBLEM - Puppet run on deployment-mediawiki06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:13:53] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:15:42] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:16:26] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:16:34] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:18:21] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:18:29] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:19:11] these all seem to fail because Wikitech itself was down, and there is some kind of caching for the Hiera mw backend [21:19:25] so when puppet code has hiera lookups, and it all does, it still fails [21:19:29] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:28:46] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:32:57] mutante: I knew puppetizing beta was a bad idea!! [21:34:06] ostriches: it turns out it was this: https://gerrit.wikimedia.org/r/#/c/327465/2 but 3 weeks ago ? [21:34:17] so maybe because puppet did _not_ run , hehe [21:34:39] i would also say putting Hiera stuff in repo is a bit more failsafe than putting it on the MW backend [21:47:01] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2917868 (10Dzahn) [21:48:23] RECOVERY - Puppet run on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:45] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:21] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:27] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:50] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:54:28] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:55:42] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:26] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:34] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [21:58:53] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T153351#2917932 (10thcipriani) [22:53:58] 10Deployment-Systems, 07I18n, 15User-greg: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2918183 (10greg) 05Open>03Resolved a:03greg This was re-enabled yesterday as planned. [23:02:25] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:02:51] Krenair: ^ oh, heh, i wonder if that is related or conincidence [23:04:21] no [23:04:32] that's a valid problem that's been going off since forever [23:04:56] we maintain a bunch of cherry-picks on top of the production branch [23:05:23] ok [23:05:27] mostly WIP stuff [23:05:39] some stuff I need to upload still [23:06:01] some stuff that is required to stop puppet being broken everywhere and isn't merged yet [23:06:10] etc. [23:10:39] Project beta-scap-eqiad build #136287: 04FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/136287/ [23:13:19] * MaxSem groans [23:13:36] "base revision is missing: HEAD" [23:13:41] gj gerrit [23:14:22] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:14:30] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:15:26] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:16:11] Project beta-scap-eqiad build #136288: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/136288/ [23:16:43] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:17:25] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:17:35] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:17:52] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:18:09] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:19:25] PROBLEM - Puppet run on deployment-mcs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:19:53] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:20:27] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:20:30] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:21:01] Project beta-update-databases-eqiad build #14021: 04FAILURE in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/14021/ [23:26:03] Project beta-scap-eqiad build #136289: 04STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/136289/ [23:32:35] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:18] Project beta-scap-eqiad build #136290: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/136290/ [23:46:23] Project beta-scap-eqiad build #136291: 04STILL FAILING in 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/136291/ [23:51:41] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:52:25] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:54:21] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:54:25] RECOVERY - Puppet run on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:54:29] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [23:54:53] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:28] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:28] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:32] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [23:57:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [23:58:08] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0]