[00:26:43] 10Gerrit, 13Patch-For-Review: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2621665 (10tom29739) [00:34:43] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [00:51:29] 10MediaWiki-Codesniffer: Provide an MWCS rule to enforce "short" type definitions - https://phabricator.wikimedia.org/T145162#2621760 (10Jdforrester-WMF) [00:51:46] 10MediaWiki-Codesniffer: Provide an MWCS rule to enforce "short" type definitions - https://phabricator.wikimedia.org/T145162#2621773 (10Jdforrester-WMF) [00:53:48] 10Gerrit, 13Patch-For-Review: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2621776 (10Paladox) I Filled upstream at https://bugs.chromium.org/p/gerrit/issues/detail?id=4526 [00:53:58] 10Gerrit, 13Patch-For-Review, 07Upstream: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2621778 (10Paladox) [02:38:57] 10MediaWiki-Codesniffer: Provide an MWCS rule to enforce "short" type definitions: int and bool, not integer and boolean - https://phabricator.wikimedia.org/T145162#2621994 (10Legoktm) [02:39:24] 10MediaWiki-Codesniffer: Provide an MWCS rule to enforce "short" type definitions: int and bool, not integer and boolean - https://phabricator.wikimedia.org/T145162#2621760 (10Legoktm) I looked upstream and the only thing I could find was https://github.com/WordPress-Coding-Standards/WordPress-Coding-Standards/i... [03:22:49] (03Abandoned) 10Awight: Workaround broken php55lint [integration/config] - 10https://gerrit.wikimedia.org/r/306050 (https://phabricator.wikimedia.org/T143598) (owner: 10Awight) [03:32:20] RECOVERY - Host deployment-parsoid05 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [03:42:29] PROBLEM - Host deployment-parsoid05 is DOWN: PING CRITICAL - Packet loss = 100% [03:48:35] (03PS3) 10Awight: Tests for DonationInterface/vendor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/304875 (https://phabricator.wikimedia.org/T143025) [04:52:38] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:32:38] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:35:54] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:06:03] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:12:54] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2622314 (10dachary) [06:15:02] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2622329 (10dachary) @Legoktm done as you suggested at https://gerrit.wikimedia.org/r/#/c/308944/ (thanks a lot for the tip ;-). Is there anything missing here ? For next time I'd be interested in the wiki... [06:15:53] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:33:49] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2622335 (10Legoktm) Great! So, the tests are run on a fresh Debian Jessie VM, which comes with Python 3.4.2. However, the tests do not have root access, so we'll need to make sure any system level depende... [06:46:00] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:42] (03PS1) 10Awight: Reuse composer-test template for SmashPig repo [integration/config] - 10https://gerrit.wikimedia.org/r/309525 [07:35:17] (03CR) 10Awight: "Some potential cleanups..." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/307543 (owner: 10Paladox) [08:03:03] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2622435 (10dachary) > those are dependencies of your program directly, not of tox/pip right? Yes, it's doing ```apt-get install mercurial fossil``` that kind of dependencies. > running against test.wik... [08:04:17] hashar: ! :) [08:04:22] good morning! [08:15:39] (03PS1) 10Addshore: Add Wikibase as a WikibaseJavaScriptApi dependancy [integration/config] - 10https://gerrit.wikimedia.org/r/309537 (https://phabricator.wikimedia.org/T145168) [08:15:57] now 100% sure if that is in the correct place :P but it feel right... [08:16:48] helllo! [08:17:10] addshore: Wikibase is rather huge isn't it ? [08:17:17] yus ;) [08:17:28] I suspect a change to WikibaseJavaScriptApi will end up trying to run the Wikibase tests [08:17:38] and I am not sure how well it is going to play [08:17:45] ooooh :/ [08:17:52] Message 'wikibase-error-remove-generic' required by 'wikibase.api.RepoApiError' must exist [08:17:52] that would be lame [08:17:52] ohh [08:17:59] yeh :/ [08:18:06] lets try ? [08:18:09] sure! :D [08:18:12] I can push that, you recheck [08:18:15] and we see what happens [08:18:23] but if it runs all of the tests then maybe we will coem up with a different solution [08:18:25] if it is too bad, rethink about it, else {done} [08:18:55] (03CR) 10Hashar: [C: 032] "Would probably try to run Wikibase tests which will most likely fail / slowdown everything. But heck lets try!" [integration/config] - 10https://gerrit.wikimedia.org/r/309537 (https://phabricator.wikimedia.org/T145168) (owner: 10Addshore) [08:19:09] babysitting some puppet deployment with joe, so poke me if I forgot to deploy :) [08:19:14] does the config repo deploy itself when things are merged yet [08:19:17] ahh, I guess not ;) [08:20:39] (03Merged) 10jenkins-bot: Add Wikibase as a WikibaseJavaScriptApi dependancy [integration/config] - 10https://gerrit.wikimedia.org/r/309537 (https://phabricator.wikimedia.org/T145168) (owner: 10Addshore) [08:22:12] * addshore pokes hashar for a deployment ;) [08:22:42] addshore: done please check mw1099 ! [08:22:54] haha [08:23:11] so ideally [08:23:21] what happens is [08:23:25] WikibaseJavaScriptApi depends on Wikibase [08:23:38] the phpunit job ends up running everything registered to the UnitTestList hook [08:23:51] when really we should probably only run tests of WikibaseJavaScriptApi [08:24:00] eg find a way to filter out registered tests in UnitTestList [08:24:18] yeh [08:26:17] (03PS6) 10Legoktm: Add composer test jobs that use PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:26:42] yeh, looks like it runs a buck load of tests.... but oh well, at least it looks like CI says yes now [08:26:55] but the idea of only running the tests for the thing being test makes lots of sense! [08:27:21] (03CR) 10Legoktm: [C: 04-1] "This touches a lot of jobs, so I'll split it into two." [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:28:55] (03PS7) 10Legoktm: Add composer test jobs that use PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:28:57] (03PS1) 10Legoktm: Add PHP 7.0 to assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/309539 [08:29:40] hashar: is it alright if I deploy changes right now? [08:29:45] (03PS8) 10Legoktm: Add composer test jobs that use PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:29:47] (03PS2) 10Legoktm: Add PHP 7.0 to assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/309539 [08:29:50] legoktm: be bold!? :) [08:30:03] legoktm: make sure to not trigger too many jobs [08:30:12] they're all experimental right now [08:30:13] the pool of instances is quite limited still [08:30:17] yeah sounsd good [08:30:30] also on one of the changes I have suggested to maybe add a php7 pipeline in zuul [08:30:53] or rename the php5 one to 'zend' or 'php' and use it for both php5 and php7 [08:31:26] if something screw up let me know and I will catch up so you while you sleep :] [08:31:59] it would be nice if we could make the php5 queue something generic that ran the set of jobs that are in gate but not test [08:32:05] since that's what it is basically for [08:34:06] (03CR) 10Legoktm: [C: 032] Add PHP 7.0 to assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/309539 (owner: 10Legoktm) [08:34:09] (03CR) 10Legoktm: "INFO:root:Updating jobs in ['config/'] (['composer-hhvm', 'composer-package-hhvm-jessie', 'composer-package-hhvm-trusty', 'composer-packag" [integration/config] - 10https://gerrit.wikimedia.org/r/309539 (owner: 10Legoktm) [08:34:26] jenkins is taking a long time to update these jobs [08:35:08] (03Merged) 10jenkins-bot: Add PHP 7.0 to assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/309539 (owner: 10Legoktm) [08:35:20] maybe because these jobs run really frequently? [08:37:30] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: The assert-phpflavor macro should probably be a shell script in integration/jenkins - https://phabricator.wikimedia.org/T145184#2622478 (10Legoktm) [08:38:33] legoktm: possibly [08:38:42] it might have to read / parse the last few builds in the history [08:38:55] hahah I already filed that bug https://phabricator.wikimedia.org/T124572 [08:38:58] also if you have an up to date jjb from integration/jenkins-job-builder, it should update the jobs in parallel [08:39:19] hmm, I should? I update whenever you send the email to qa@ :P [08:39:31] so should be fine :) [08:39:33] I'm on6fcaf39b1c4cfd45d76c4453c315c2fdb8509121 [08:39:49] 6fcaf39b1c4cfd45d76c4453c315c2fdb8509121 Ifc979cdb8e40ad31debff04f745649f23f7ef91a [08:39:51] sounds right [08:40:49] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Move assert-phpflavor macro into a bash script in integration/jenkins - https://phabricator.wikimedia.org/T124572#2622496 (10Legoktm) [08:40:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: The assert-phpflavor macro should probably be a shell script in integration/jenkins - https://phabricator.wikimedia.org/T145184#2622498 (10Legoktm) [08:41:14] so [08:41:21] hashar: hmm https://paste.fedoraproject.org/424415/41047414/raw/ [08:41:25] jenkins-jobs update --workers=4 [08:41:29] it default to 1 [08:41:32] aha! [08:41:33] set to 0 for auto detection [08:41:47] should I use 0 then? [08:41:50] and maybe that is a setting available in jenkins_jobs.ini [08:41:53] so yeah 0 maybe [08:42:07] #!/bin/bash [08:42:08] echo "Deploying '$@'"; [08:42:08] /home/km/python/bin/jenkins-jobs --conf etc/jenkins_jobs.ini update config/ $@ [08:42:15] that's my script to deploy stuff :) [08:42:17] I never used it on our setup [08:42:41] parsoidsvc-hhvm-parsertests-jessie also 503'd [08:42:56] wait no, different traceback [08:43:19] okay, I found a bug in jjb [08:43:37] \O/ [08:43:51] so the 503 are sometime due to Jenkins timing out while reaading the build history [08:43:56] in such a case I just retry [08:44:16] Jenkins is processing the build history in the background, and eventually the http threads times out before the job is complete [08:44:27] on next try, the background job has completed and the build history is in the Jenkins cache [08:44:30] so the request pass [08:45:00] the bug I found is that after jjb encounters a 503, it will continue updating the rest of the jobs, but it doesn't save them to the cache. so when you re-deploy the jobs, it will start with the one that failed, and then re-deploy everything after that failed one. https://phabricator.wikimedia.org/P4014 [08:45:40] ohh [08:46:02] urllib2.HTTPError: HTTP Error 503: Backend fetch failed [08:46:02] INFO:jenkins_jobs.builder:Cache saved [08:46:34] :( [08:47:33] (03CR) 10Legoktm: [C: 032] "\o/" [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:48:31] (03Merged) 10jenkins-bot: Add composer test jobs that use PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/309048 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [08:49:00] !log deploying https://gerrit.wikimedia.org/r/309048 [08:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:51:52] oh sigh [08:51:57] need to label the nodes first [08:53:20] phuedx|zzZ: apologies for the delay, I was busy with other stuff, working on https://phabricator.wikimedia.org/T144912 [08:53:22] !log added phpflavor-php70 label to integration-slave-jessie-100[1-5] [08:53:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:54:48] https://integration.wikimedia.org/ci/job/composer-php70/1/console [08:55:16] wooo :D [08:55:31] :D [08:55:51] oh also [08:56:06] yesterday we have noticed that the php53lint job failed due to it invoking: $PHP_BIN -v [08:56:17] I had Zuul to inject the parameter for the *php53* jobs [08:56:20] and looks all fine [08:56:35] ah yeah, I noticed that [08:57:32] and appareltny the phplint jobs are the last still using PHP_BIN directly [08:58:37] (03PS1) 10Legoktm: Use plain `php` in phplint macro [integration/config] - 10https://gerrit.wikimedia.org/r/309541 [08:58:57] I grepped for PHP_BIN in jjb/ and didn't see anything else [08:59:13] yeah [08:59:15] sounds good enough [09:00:38] is there a task about expanding CI capacity that I can block the composer-test on php7 one on? [09:00:48] yeah [09:01:03] I am slowing moving jobs back to nodepool [09:01:12] with chase closely monitoring labs infra [09:01:32] https://phabricator.wikimedia.org/T133911 "Bump quota of Nodepool instances (contintcloud tenant)" [09:02:08] oh also, I'm not sure if you've seen https://phabricator.wikimedia.org/T142457 ? [09:02:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:03:05] I want to have one job that sets PHP_BIN, runs composer install && composer test, switches PHP_BIN, and repeats for 5.5, 5.6, hhvm, and 7.0 [09:03:46] (03CR) 10Legoktm: [C: 032] Use plain `php` in phplint macro [integration/config] - 10https://gerrit.wikimedia.org/r/309541 (owner: 10Legoktm) [09:04:39] 10Continuous-Integration-Config: Combine composer-php55 and composer-hhvm jobs - https://phabricator.wikimedia.org/T142457#2622521 (10Legoktm) Once {T144959} is done, we can have one job that runs 5.5, 5.6, 7.0, and hhvm all on jessie. [09:04:51] (03Merged) 10jenkins-bot: Use plain `php` in phplint macro [integration/config] - 10https://gerrit.wikimedia.org/r/309541 (owner: 10Legoktm) [09:05:02] 10Continuous-Integration-Infrastructure, 06Labs: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2622524 (10hashar) I am closing this task. The quota used to be 20 until July 4th when it got lowered down to 10 in an emergency due to wmflabs being full. We ge... [09:05:17] 05Continuous-Integration-Scaling, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2622528 (10hashar) [09:05:19] 10Continuous-Integration-Infrastructure, 06Labs: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2622530 (10hashar) [09:05:38] legoktm: I will probably remove a bunch of instances from "integration" project [09:05:42] to free up capacity on labs [09:06:32] legoktm: for running multiple zend version, a makefile would probably work [09:06:58] a makefile for what? [09:07:17] to iterate over different PHP_BIN vers? [09:07:51] would also need to have 5.5 5.6 and 7.0 on Jessie [09:08:01] oh, I was just going to do it in jjb [09:08:07] with 5.5 apparently available from the 3rd party apt repo you have found [09:08:09] oh [09:08:18] or jjb yeah easier :] [09:08:21] make is scary [09:08:30] 5.6 is the default in jessie, and 7.0 and 5.5 are in the external repo [09:08:45] :) [09:08:46] also have you managed to force refresh a nodepool snapshot ? [09:08:50] no! [09:08:50] you asked about it a few days ago [09:08:53] :( [09:08:56] I was waiting for you to be around :P [09:09:00] is now a good time? [09:09:04] we should pair that next week in your morning [09:09:39] isn't it like 2am in SF ? :D [09:09:54] yes, but I prefer staying up late to waking up early :P [09:10:20] well we can do now. Should take half an hour total [09:10:48] ok [09:10:52] so I'm going to follow https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot [09:10:58] yeah [09:11:13] and I run that from labnodepool1001 correct? [09:11:17] so some pristine image is build manually and pushed to openstack labs as ci-jessie-wikimedia [09:11:24] what "nodepool image-update wmflabs-eqiad ci-jessie-wikimedia" does is [09:11:30] boot an instance out of tha tpristine image [09:11:36] ssh to it to run a setup_node.sh script [09:11:42] which runs puppet apply dib/puppet/ciiimage.pp [09:11:56] since the pristine image already had that puppet script realized, it has most of the packages / stuff already [09:12:12] thus update is merely to catch up with changes that happened in puppet.git [09:12:25] once the setup_node.sh / puppet apply is completed [09:12:42] nodepool ask openstack to snapshot the instance and make it an image [09:12:55] it then update the list of images as seen in "nodepool image-list" [09:13:02] and from now, the new instances will use that new images [09:13:18] so when we make puppet changes in the contint role, in addition to it being deployed via the integration-puppetmaster, we also have to refresh the nodepool snapshot? [09:13:20] nodepool takes a copy of the previous snapshot instead things get weird, and delete the n-2 version [09:13:29] yeah correct legoktm [09:13:58] hm, so there's not really a good way to view a full diff of everything that is going to change [09:14:03] nodepool has support to run a command just after an instance as spawned, to do some last minute commands before the instance is added to jenkins [09:14:12] so it would catch up with puppet changes on each instance spawn [09:14:37] nodepool@labnodepool1001:~$ git -C /etc/nodepool/wikimedia/ pull [09:14:37] ... [09:14:37] Updating fc3453e..69ff4d3 [09:14:39] but: (a) that slightly slowdown the time to get the instance ready (b) if puppet.git is broken, it will discard the instance and spawn a new one in a loop => scary [09:15:14] and now I run $ nodepool image-update wmflabs-eqiad ci-jessie-wikimedia; and wait? [09:15:19] yeah [09:15:41] should spurt out the setup_nodes.sh stdout/stderr to the terminal [09:15:50] ok, so no other log file I should watch? [09:15:56] then at some point it will say that ./setup_node.sh ci-jessie-image-123457 completed [09:16:13] then it takes a snapshot and interact with the openstack API to upload the image [09:16:24] err not upload: to wait for the image to be ready [09:16:40] afaik everything is dumped to the terminal [09:16:51] 2016-09-09 09:16:38,127 INFO paramiko.transport: Connected (version 2.0, client OpenSSH_6.7p1) [09:16:51] /usr/lib/python2.7/dist-packages/paramiko/client.py:580: UserWarning: Unknown ssh-rsa host key for 10.68.23.233: 65eec466cde1fc1929676061141b2de7 [09:16:51] (key.get_name(), hostname, hexlify(key.get_fingerprint()))) [09:16:56] yeah "normal" [09:17:03] right, because it's a new instance? [09:17:09] nodepool ssh to the instance [09:17:15] and doesn't know about the host ssh fingerprint [09:17:19] but it hapilly ignore it [09:17:26] though paramiko still raise an INFO message [09:17:30] uhh [09:17:31] 2016-09-09 09:16:48,335 INFO paramiko.transport: Connected (version 2.0, client OpenSSH_6.7p1) [09:17:31] 2016-09-09 09:16:48,490 INFO paramiko.transport: Authentication (publickey) failed. [09:17:31] 2016-09-09 09:16:48,659 INFO paramiko.transport: Authentication (publickey) failed. [09:17:32] 2016-09-09 09:16:48,660 INFO nodepool.utils: Password auth exception. Try number 1... [09:17:37] yeah that part is lame [09:17:43] it loops over a hardcoded set of username [09:17:59] something like: root , redhat_cloud_user, cloud-init, debian etc [09:18:12] it takes a few iterations until it catch up [09:18:13] heh [09:18:17] yeah, moving forward now [09:18:21] something like that [09:18:45] in another term [09:18:50] if you look at " nodepool image-list": [09:18:58] so is apt-get update && upgrade run when the image is refreshed, or on individual instance creation? [09:19:01] | 999 | wmflabs-eqiad | ci-jessie-wikimedia | ci-jessie-wikimedia-1473412532 | building | 0.05 | [09:19:19] oh, it just ran apt-get dist-upgrade [09:19:28] update/upgrade are done via nodepool update [09:19:36] when instance boots, nothing is run at all [09:20:01] the scripts being used are defined in modules/nodepool/templates/nodepool.yaml.erb [09:20:08] setup: setup_node.sh [09:20:11] used by nodepool update [09:20:19] ready-script: ready.sh [09:20:26] used when booting an instance meant to be added as a slave [09:20:32] so if there's say, a PHP security update, we need to refresh the image for it to be picked up? [09:21:02] both scripts are injected from labnodepool1001.eqiad.wmnet /etc/nodepool/wikimedia/nodepool/scripts/ [09:21:19] so in theory, i think you can live hack the scripts on labnodepool1001 and your hack will be injected in the instance [09:21:24] 2016-09-09 09:19:46,046 INFO nodepool.SnapshotImageUpdater: Image ci-jessie-wikimedia-1473412532 in wmflabs-eqiad is ready [09:21:30] yeah gotta refresh the image [09:21:32] it's done! [09:21:47] the setup.sh script (for the sanpshot) is run with the debian user that has sudo root [09:21:56] but the ready script is run as the jenkins user which does not have sudo [09:22:16] so nodepool image-list no longer shows ID 994, and that's intentional? [09:22:16] but we could grant sudo to jenkins user, and have the ready script to do apt-get upgrade etc, then drop the sudo rule [09:22:22] yeah [09:22:26] you got a new snapshot [09:22:30] so the n-2 version got deleted [09:23:01] ci-jessie-wikimedia-1473344040 from 19.04 hours ago is till around though [09:23:09] so if the new snapshot turns out to be an issue [09:23:20] gotta delete the new one, and nodepool will falback to the n-1 one [09:23:28] (eg it keeps up to 2 snapshots) [09:23:31] that is the lame rollback [09:23:46] then "nodepool list [09:23:46] " [09:23:48] ah okay [09:23:50] give you the list of instances [09:24:00] the jessie ones have been booted with the previous snapshot [09:24:11] how can I tell that they're using the old snapshot? [09:24:17] so you cant know whether everything is going fine until instances using the new snapshot have been created [09:24:20] just Age? [09:24:23] yeah age [09:24:31] "nodepool list" output is bad [09:24:35] it lacks the image id / name [09:24:45] should I delete the jessie ones using nodepool delete so we can test the image? [09:24:59] but if you use "openstack server show ci-jessie-370147" you will get more details [09:25:39] or Horizon [09:25:44] * legoktm nods [09:25:52] openstack server show ci-jessie-wikimedia-370147 [09:25:53] | image | ci-jessie-wikimedia-1473344040 (6b06b5eb-eb39-452e-a5fa-68e05799f323) | [09:26:04] horizon list of instances show the image name iirc [09:27:17] sometime [09:27:28] I will "nodepool delete" the outdated instances [09:27:39] ok [09:27:39] and it will boot new ones using the newer snapshot [09:28:00] I assume I should !log something? [09:28:02] and https://horizon.wikimedia.org/project/instances/ does show the image name being used [09:28:04] yeah [09:28:24] I usually !log the command that state ci-jessie-wikimedia-1473344040 is ready [09:28:52] example from yesterday: 13:13 Image ci-jessie-wikimedia-1473253681 in wmflabs-eqiad is ready , has php7 packages. T144872 [09:29:11] which poke the task informing subscriber that in theory nodepool/tests can start using php7 [09:29:19] then [09:29:21] scary thing [09:29:30] nodepool automatically update the snapshot everyday at 14:14 UTC [09:29:39] oh, it does? [09:29:55] so it keeps up with apt-get upgrade and puppet.git changes [09:29:57] if that fails [09:30:05] it will just stick to last known snapshot image [09:30:21] which will keep aging [09:30:26] until puppet is fixed [09:30:35] so my puppet change from a few days ago most likely was already deployed? [09:30:43] !log Image ci-jessie-wikimedia-1473412532 in wmflabs-eqiad is ready [09:30:43] if it got merged? yeah [09:30:46] on a schedule [09:30:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:30:53] ok [09:30:59] so this was just for fun then :P [09:31:00] else you get ops to merge it [09:31:06] hook on labnodepool and force refresh the snapshot [09:31:25] at least you are trained and get your "update nodepool snapshot achievement" unlocked! [09:31:34] :) thanks for walking me through it [09:31:35] building images is scarier [09:32:01] the short idea is using debootstrap to populate an entirely fresh system [09:32:04] chroot into it [09:32:20] run diskimage-builder which is more or less a shell based equivalent of puppet/ansible [09:32:37] that runs a **** ton of commands to tweak the filesystm / add packages / run puppet in the chroot [09:32:49] then eventually craft a qemu image out of the resulting chroot [09:33:12] qemu image which I then manually uploaded to wmflabs as new reference image (eg ci-jessie-wikimedia ) [09:33:28] and then "nodepool update ..." to create a new snapshot bsed on that new image [09:33:45] it is rarely needed luckily [09:33:50] huh, that's scary, but also pretty cool [09:33:53] yeah [09:33:59] labs team is using a similar process [09:34:37] using modules/labs_bootstrapvz for Jessie [09:34:43] modules/labs_vmbuilder/ for Trusty [09:35:01] which must be done in a labs instance because it copies a bunch of files from the instance /etc to the image [09:35:07] such as ldap credentials / ssh config etc [09:35:17] so we have three tools to build images :] [09:36:56] well [09:37:04] gonna try to deb package a python module [09:37:52] 10Continuous-Integration-Infrastructure: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#2622580 (10Legoktm) [09:38:12] python debs should be pretty easy, debhelper does most if it [09:38:30] I will email wikitech-l about the jobs being experimental and then sleep :) [09:38:46] sounds great! [09:38:50] thanks a ton for that effort [09:38:56] I am sure Erik B. will be happy [09:39:05] he was asking about it to run some php linter of some sort [09:39:23] yeah, that's https://phabricator.wikimedia.org/T132636 [09:39:40] but php-ast isn't in the repo so I need to ask the maintainer if they'd be willing to add it [09:39:49] I also have to ask them for a few other modules that we are missing [09:40:48] 10Continuous-Integration-Infrastructure: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#2622583 (10hashar) >>! In T144964#2616200, @Paladox wrote: > Ok, but doint we have two Jessie instances we could use? Indeed. @Legoktm as marked the few permanent Jessie slaves with a label: https:... [09:42:05] 10Continuous-Integration-Infrastructure: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#2622588 (10Legoktm) [09:42:07] 10Continuous-Integration-Config, 13Patch-For-Review: Create composer-php70 job - https://phabricator.wikimedia.org/T144961#2622584 (10Legoktm) 05Open>03stalled These are now deployed in the experimental pipeline - we're now waiting for 1) some people to test out their extensions and make sure they pass 2)... [09:42:52] ah https://deb.sury.org/ mentions that the work is upstreamed to Debian unstable quite nice [09:43:11] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622601 (10zeljkofilipin) `#get_wikitext` works fine when targeting my local mediawiki-vagrant machine. ``` $ irb irb(m... [09:44:01] yeah, he is one of the debian PHP maintainers :) [09:44:29] maybe Erik B. / Greg has some budget to sponsor the build of php7-ast :] [09:45:19] heh [09:45:28] well it is in debian unstable [09:45:41] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:45:52] I'll probably email him tomorrow (well today now) about adding more extensions [09:46:16] legoktm: feel free to cc me [09:46:28] will do [09:46:39] * hashar discovers Patreon [09:46:43] https://www.patreon.com/oerdnj [09:46:50] my bait they take a 15% fee for everything [09:50:49] Patreon takes 5% of successfully-processed payments to keep the lights on. [09:51:00] + other fees due to payment processors [09:51:10] 5% sounds fair [09:52:30] https://www.patreon.com/rubinreport get 2700 supporter for 22k/months [09:56:58] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:08:41] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622643 (10zeljkofilipin) Running failed scenario (`diff.feature:5`) on master runs fine when targeting my local mediawik... [10:13:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:15:33] 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations, 13Patch-For-Review: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2479747 (10elukey) @Dereckson,... [10:19:07] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622670 (10zeljkofilipin) Everything runs fine locally even with commit [[ https://gerrit.wikimedia.org/r/#/c/308956 | 30... [10:31:57] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:40:32] 10Gerrit, 13Patch-For-Review, 07Upstream: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2622711 (10Aklapper) p:05High>03Low >>! In T145130#2621776, @Paladox wrote: > I Filled upstream at https://bugs.chromium.org/p/gerrit/issues/... [10:40:36] 10Gerrit, 13Patch-For-Review, 07Upstream: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2622713 (10Aklapper) >>! In T145130#2620818, @Paladox wrote: > This affects users who use Microsoft Edge so setting as high priority. @Paladox:... [10:43:19] 05Continuous-Integration-Scaling, 06Operations, 07Nodepool, 07WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#2622714 (10hashar) The commit just after the version we run ( 9e2937cedc9360e45c70f9038dca6dd44b0c6460 ) just ha... [10:43:36] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622715 (10zeljkofilipin) [[ https://phabricator.wikimedia.org/diffusion/CICF/browse/master/jjb/mediawiki-extensions.yaml... [10:43:46] hashar: ^ [10:44:05] 10Continuous-Integration-Infrastructure, 07Nodepool: Update Nodepool to catch up with upstream master branch - https://phabricator.wikimedia.org/T144601#2622716 (10hashar) From T107267#2622714 I think I will aim for Nodepool `0.2.0` with shade `0.16.0`. A test instance will surely help. [10:44:23] yeah was kind of busy with nodepool / shade versions [10:44:29] ah [10:44:40] zeljkof: yeah #wikimedia-mobile folks talked about it earlier this week [10:44:46] I have happilly redirected them to you :D [10:44:51] hashar: :D [10:45:12] I am working on it, but not familiar with the mwext-mw-selenium job [10:45:22] looks like an env variable is not set up correctly somewhow [10:45:32] well [10:45:50] the local mediawiki install does not have path rewriting [10:46:10] iirc [10:46:30] the $WORKSPACE is symlinked under /srv/localvhost [10:46:55] ln -s /mnt/jenkins-workspace/workspace/mwext-mw-selenium/src /srv/localhost-worker/jenkins-mwext-mw-selenium-9801 [10:47:21] and the curl command reflect that: http://localhost:9412/jenkins-mwext-mw-selenium-9801/index.php/Special:BlankPage [10:47:35] and [10:47:43] the job does export MEDIAWIKI_URL=http://localhost:9412/jenkins-mwext-mw-selenium-9801/index.php/ [10:48:01] (note the trailing slash) [10:48:06] but export MEDIAWIKI_API_URL=http://localhost:9412/jenkins-mwext-mw-selenium-9801/api.php (no trailing slash) [10:48:18] ah [10:48:22] so at first: I have no idea why MEDIAWIKI_URL has a trailing slash [10:48:27] might have been done intentionally [10:48:30] should it also be /w/api.php [10:48:33] or it is a mistake [10:48:52] and I can imagine that mediawiki_selenium eventually get confused [10:48:56] everything seems to be working fine, except this one api call o.O [10:49:13] seeing /index.php/ it would think that is the root of the repo and appends /w/index.php to get the full path [10:49:22] and I am not sure what MEDIAWIKI_URL is supposed to be [10:49:38] MediawikiApi::Client#get_wikitext right ? [10:49:44] is that in mediawiki_ruby ? [10:50:44] hashar: I am not sure that is correct anyway, why isn't the job using rake? [10:50:56] ok, reading the code, just found mw-selenium-setup.sh [10:51:34] copy pasting to task [10:52:10] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622723 (10zeljkofilipin) [[ https://phabricator.wikimedia.org/diffusion/CIJE/browse/master/bin/mw-selenium-setup.sh | mw... [10:52:20] thanks [10:53:31] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:55:39] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622724 (10hashar) The job clone the repos under $WORKSPACE/src then we publish that to Apache with a symlink: ln -... [10:55:43] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:01] zeljkof: the trailing slash is probably needed / a hack [10:56:10] the question is why MediawikiApi::Client#get_wikitext uses MEDIAWIKI_URL ? [10:58:33] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622725 (10hashar) [10:58:40] Using mediawiki_api 0.7.0 [10:59:07] * 2f83f58 - Support automatic redirects (5 weeks ago) [10:59:17] probably not related [10:59:33] also mediawiki/ruby/api.git lacks a git tag for 0.7.0 :] [10:59:44] really? [10:59:55] I think I was releasing it via "rake release" [10:59:56] yeah :) [11:00:01] it should tag automagically [11:00:03] lemme check [11:00:04] and 0.6.0 got tagged v0.6.0 [11:00:13] when previous vers do not have a leading 'v' [11:00:15] yes, that is the way rake release tags it [11:00:16] but I am nitpicking hehe [11:01:35] 10Browser-Tests-Infrastructure, 10Wikidata, 07Browser-Tests: Run Wikibase browser tests on gerrit triggered with keyword - https://phabricator.wikimedia.org/T145190#2622727 (10Jonas) [11:01:58] probably Dan released it then, he made the last commit, and forgot to release via rake [11:02:26] tagged [11:02:34] * 27fb736 - (HEAD -> master, tag: v0.7.0, gerrit/master, gerrit/HEAD) Release minor version 0.7.0 (5 weeks ago) [11:05:15] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622740 (10zeljkofilipin) [[ https://phabricator.wikimedia.org/diffusion/CIJE/browse/master/bin/mw-set-env-mw-selenium.sh... [11:07:03] hashar: I am completely confused how would api calls work fine but fail on this one instance :| [11:07:13] and I am not sure how to test it locally [11:07:22] if the patch is rechecked does it work ? [11:07:27] I mean , running on another slave [11:07:31] will try [11:07:40] it did run twice, and failed both times [11:07:45] hey v1.7.2 hasn't been tagged for mediawiki/Selenium [11:07:46] tagging [11:08:20] !log Added git tag for latest versions of mediawiki/selenium and mediawiki/ruby/api [11:08:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:08:41] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622741 (10hashar) ``` lang=ruby def get_wikitext(title) @conn.get '/w/index.php', action: 'raw', title: title... [11:14:40] hashar: could it be that nobody ever used get_wikitext in jenkins? [11:14:49] I am reading the code, it was added 2 years ago [11:14:55] and looks like never refactored since [11:18:36] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622759 (10zeljkofilipin) [[ https://phabricator.wikimedia.org/diffusion/MRUB/browse/master/lib/mediawiki_api/client.rb;2... [11:19:03] zeljkof: that is a possibility yeah [11:19:22] maybe try writing a very basic example [11:19:31] that would be run with MEDIAWIKI_URL and MEDIAWIKI_API_URL [11:19:50] then try to invoke get_wikitext() ? [11:19:58] inspecting the @conn object [11:20:15] it has an url_prefix set to index.php [11:21:26] so to me it looks like mediawiki_selenium creates the Api Helper with MEDIAWIKI_URL [11:21:55] looks like the job does not use a lot of new features in mw-selenium [11:22:04] calls to the api ends up using either the url from MEDIAWIKI_API_URL or some heuristic to figure out /w/api.php from MEDIAWIKI_URL [11:22:07] and gets into trouble because of that [11:22:14] but it looks like mediawiki/selenium initialize the API helper with MEDIAWIKI_URL [11:22:40] and that get_client uses the underlying connections directly ( @conn.get ) which is set with the url prefix MEDIAWIKI_URL ==> fail [11:22:42] then [11:22:53] I am not sure how the other methods from mediawiki/api manage to hit the proper url [11:23:18] they don't use the explicit @conn.get '/w/index.php' [11:24:09] but #action [11:24:11] action(:delete, title: title, reason: reason) [11:24:42] I am refreshing my mw-api, will figure it out [11:24:42] yeah [11:25:01] thanks a lot [11:25:07] maybe try with writing a very basic spec that reproduce the issue [11:25:08] now I know where to look [11:25:10] will help debugging [11:25:18] but most probably get_wikitext() is broken [11:25:33] not sure if we can easily find out whether it is used anywhere [11:26:06] been added in https://gerrit.wikimedia.org/r/#/c/124384/ [11:26:08] by manybubbles [11:26:13] so maybe elastica / cirrussearch [11:33:54] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622775 (10hashar) That method has been added in April 2014 apparently for CirrusSearch by https://gerrit.wikimedia.org/... [11:33:56] zeljkof: that is all I have [11:34:14] zeljkof: but I have found the commit in cirrusSearch that rely on get_wiki_text() [11:44:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:49:19] hashar: thanks, I am also grepping ruby repos I have locally [11:49:27] and testing the api [12:05:03] 10Gerrit, 13Patch-For-Review, 07Upstream: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2622851 (10Paladox) The reason I only have a patch on our gerrit is because it is a workaround and not a fix. But isent a browser that users use... [12:07:20] 10Continuous-Integration-Config, 13Patch-For-Review: Create composer-php70 job - https://phabricator.wikimedia.org/T144961#2622854 (10Paladox) Thankyou, I guess we should send an email encouraging users to check experimental and report back here with the results of the php 7 test :) [12:07:41] (03PS1) 10Paladox: Add composer-php70 as a experimental test to mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) [12:08:42] 03Scap3, 10Parsoid, 06Services, 15User-mobrovac: Enable Scap3 config deploys for Parsoid - https://phabricator.wikimedia.org/T144596#2622867 (10mobrovac) [12:08:45] 03Scap3, 10Citoid, 06Services, 10VisualEditor, and 2 others: Enable Scap3 config deploys for Citoid - https://phabricator.wikimedia.org/T144597#2622866 (10mobrovac) [12:08:49] 03Scap3, 06Services, 10service-runner, 10service-template-node, 15User-mobrovac: Enable config deploys for service::node services - https://phabricator.wikimedia.org/T144542#2622860 (10mobrovac) 05Open>03Resolved Merged, resolving. Will deal with individual services in follow-up tasks. [12:08:52] 03Scap3, 10ChangeProp, 10EventBus, 06Services, 15User-mobrovac: Enable Scap config deploys for Change Propagation - https://phabricator.wikimedia.org/T144595#2622868 (10mobrovac) [12:09:37] 10Browser-Tests-Infrastructure, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2622869 (10zeljkofilipin) The only two places where #get_wikitext is used are: ``` CirrusSearch/tests/browser/features/s... [12:23:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:24:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:45:39] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Wikidata: Run Wikibase browser tests on gerrit triggered with keyword - https://phabricator.wikimedia.org/T145190#2622891 (10zeljkofilipin) [12:49:50] 03Scap3: Scap fails to force-deploy the config - https://phabricator.wikimedia.org/T145194#2622895 (10mobrovac) [12:50:12] 03Scap3, 10Mathoid, 06Services, 13Patch-For-Review, 15User-mobrovac: Enable Scap3 config deploys for Mathoid - https://phabricator.wikimedia.org/T144755#2609180 (10mobrovac) [12:50:13] 03Scap3: Scap fails to force-deploy the config - https://phabricator.wikimedia.org/T145194#2622910 (10mobrovac) [13:03:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [13:04:45] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:19:05] 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations, 13Patch-For-Review: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2622982 (10Dereckson) The T1377... [13:27:19] 10Gerrit, 13Patch-For-Review, 07Upstream: Gerrit in Microsoft Edge doesn't display the git commands in the download box - https://phabricator.wikimedia.org/T145130#2622998 (10Aklapper) There is always [[ https://www.mediawiki.org/wiki/Gerrit/Tutorial | `git review -d 123456` ]] as a workaround, no matter whi... [13:44:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:12] hashar: apparently #get_wikitext is not using the api, but getting the text directly from the web interface [13:46:35] and the docs about how to do it via the api seem to be incorrect [13:46:36] https://www.mediawiki.org/wiki/API:FAQ#get_the_content_of_a_page_.28wikitext.29.3F [13:46:59] this does not return page wikitext, but the docs say it does https://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=Main_Page [13:47:24] I see, it is not the wikitext, but the data about the page [13:47:34] if only I could read :| [13:59:25] yeah get_wikitext uses title=Foobar&action=raw [13:59:53] something like that [14:01:25] then bah [14:01:33] https://commons.wikimedia.org/wiki/User:Legoktm works [14:01:42] https://commons.wikimedia.org/w/index.php?title=User:Legoktm --> 404 [14:01:44] I am rusty [14:02:51] curl 'https://commons.wikimedia.org/w/index.php?title=User:Hashar&action=raw' [14:02:51] works [14:03:57] that's how the mw-api works now [14:04:10] the big q being [14:04:19] why does cirrussearch has the proper url [14:04:25] is there really no way to get wikitext from the api!? [14:04:28] while mobilefrontend ends up with a prefix [14:04:40] ohh [14:04:47] you wanna stop using action=raw [14:04:57] I am not familiar with the api :( [14:05:44] neither am I [14:05:50] I think I am getting there... [14:05:53] What are you trying to get? [14:05:56] * ostriches yawns [14:05:58] morning [14:05:59] https://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=Main%20Page&rvprop=timestamp|user|comment|content [14:06:04] ostriches: morning! [14:06:14] I am trying to get wikitext using the api [14:06:50] You're looking in the right place. [14:06:59] query / revisions / titles [14:07:06] rvprop content should have it [14:07:08] zeljkof: https://commons.wikimedia.org/w/api.php?action=query&prop=revisions&titles=User:Hashar&rvprop=content [14:07:29] and https://commons.wikimedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&titles=API%7CMain+Page&rvprop=timestamp%7Cuser%7Ccomment%7Ccontent is quite nice [14:08:43] thanks hashar and ostriches, I think I am getting there [14:09:05] the apisandbox is quite good [14:23:40] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:53:29] ah [14:53:31] puppet flap [14:57:19] puppet master races with the auto rebase script :( [15:03:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:05:35] 05Gerrit-Migration, 10Differential, 10Phabricator, 13Patch-For-Review: Enable differential.allow-self-accept in phabricator - https://phabricator.wikimedia.org/T131622#2172920 (10Ottomata) +1 I'd like this too please. [15:09:06] off [15:09:11] might be back later this evening [15:09:37] 05Gerrit-Migration, 10Differential, 10Phabricator, 13Patch-For-Review: Enable differential.allow-self-accept in phabricator - https://phabricator.wikimedia.org/T131622#2623143 (10Paladox) I guess we should really implement a way so we have a whitelist like we do on gerrit. We have +1 for users and +2 for... [15:36:57] ori: About? [15:37:05] ostriches: hey [15:37:56] Heya. I'm looking at multiversion code and I spotted MWMinimalScriptInit that you introduced for mwrepl. [15:38:10] I'm curious what the exact use case is, and why the MWScript entrypoint doesn't work for you [15:38:27] (background: trying to trim down extraneous multiversion code and limit entry points) [15:43:53] ci people, it looks like jenkins is rejecting all puppet patches now, due to a failure to import 'monkey' [15:44:01] does that ring a bell for anyone? [15:45:23] https://integration.wikimedia.org/ci/job/operations-puppet-tox-jessie/4844/console [15:45:30] 15:41:31 Collecting nose [15:45:30] 15:41:31 Downloading nose-1.3.7-py2-none-any.whl (154kB) [15:45:30] 15:41:31 Collecting PyYAML (from -r modules/admin/data/requirements.txt (line 1)) [15:45:30] 15:41:31 Downloading PyYAML-3.12.tar.gz (253kB) [15:45:30] 15:41:31 Could not import setuptools which is required to install from a source distribution. [15:45:31] 15:41:31 Traceback (most recent call last): [15:47:09] hasharAway ^^ [15:48:06] ostriches: let me dig through the git history to remind myself [15:48:36] I think it's so mwrepl can do an hhvm debugger session with base MW entry defined. [15:49:08] (which I'm curious if mwrepl can become a wrapper for eval.php then?) [15:50:04] well, the HHVM debugger has a lot of features that eval.php does not [15:50:52] Yeah, but I'm curious if we could use the hhvm debugger to open the script for us? [15:50:57] Something like.... [15:51:11] it was intended for mwrepl (which is indeed using it), but I'm trying to remember why MWScript wouldn't do [15:51:33] $PROLOGUE /usr/bin/hhvm --mode=debug --debug-extension="path/to/mwscript.php ${MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE}/php/maintenance/eval.php" [15:51:34] andrewbogott it should work now [15:51:42] after a recheck it looks like it is passing now [15:51:43] :) [15:52:10] well, couple of issues [15:52:22] 10Deployment-Systems, 03Scap3, 10scap, 06Operations, 13Patch-For-Review: Make keyholder work with systemd - https://phabricator.wikimedia.org/T144043#2623263 (10AlexMonk-WMF) [15:52:39] one, what value is eval.php adding that isn't already provided by mwscript.php? the repl functionality is all provided by the hhvm debugger, so all that is needed is a way to bootstrap mediawiki's code [15:52:51] Ahhh, ok I see it now [15:53:11] mwscript is just the wrapper for multiversion bootstrap [15:53:20] two, mwscript.php picks a wiki by parsing $argv [15:53:21] You could actually still use commandLine.inc there [15:53:44] so I probably introduced MWMinimalScriptInit to have code that doesn't assume anything about the structure and format of the command-line invocation [15:55:03] well, you need to know which wiki to load [15:55:16] so you still need something wrapping commandLine.inc [15:55:22] I think we can tweak MWScript easy enough to take an env variable [15:55:36] Give mwscript the env variable instead of --wiki [15:55:48] Then pass commandLine.inc to mwscript [15:55:53] (lemme see if that works, it should) [15:56:31] Yep, commandLine.inc will work here [15:56:50] well, that's what MWMinimalScriptInit.php calls [15:57:09] 03Scap3: Scap fails to force-deploy the config - https://phabricator.wikimedia.org/T145194#2623286 (10mobrovac) The problem seems to be the fact that Scap3 tries to `sudo` into the deployment user, which is the same as the one running the script. Alas, the deployment user cannot do `sudo -i ${deployment_user}`.... [15:57:57] in general tho, if you can see a way to de-duplicate bootstrapping code, I'm all for it [15:57:58] 03Scap3: Scap fails to force-deploy the config - https://phabricator.wikimedia.org/T145194#2623303 (10mobrovac) Confirmation of the above: ``` deploy-service@deployment-mathoid:~$ sudo -i deploy-service scap deploy-local --help We trust you have received the usual lecture from the local System Administrator. I... [16:00:31] ori: Ah, I see a completely less convoluted way to do it [16:00:34] Will work up a patch [16:22:10] ori: https://gerrit.wikimedia.org/r/#/c/309601/ gets at what I'm thinking [16:22:34] After reading it closely, as long as mwrepl sets MW_WIKI, Multiversion should use that and skip the errors about no --wiki [16:23:22] mwrepl can set MW_BOOTSTRAP_ONLY as well, which would completely obsolete MWMinimalScriptInit [16:23:32] Yeah that's the plan [16:23:44] But multiversion isn't liking me hmmmm [16:24:49] This parameter wrangling sucks. [16:25:30] zeljkof: thanks for that patch -- i was probably going to do the same thing [16:25:57] the get_wikitext patch, i mean [16:27:01] Ah ok there we go. [16:28:21] ori: Ok yeah that looks like it'll work from some testing on tin [16:36:14] Actually, I don't even think my patch is needed. [16:36:26] I think mwrepl can just to MWScript.php commandLine.inc [16:36:33] Long as MW_WIKI is set first [16:39:36] ori: https://gerrit.wikimedia.org/r/#/c/309605/ should *just work* without any other changes even needed. [16:40:08] have you tried it? [16:40:39] Errr, not quite working [16:40:42] Thought it would.... [16:41:05] Ahhhh, debug-extension doesn't like a space [16:41:26] Ah, gotta be outside the quote [16:41:28] Yes it works! [16:42:06] Almost. [16:42:10] Dammit I will get this right! [16:42:11] :D [16:44:38] 05Gerrit-Migration, 10Differential, 10Phabricator, 13Patch-For-Review: Enable differential.allow-self-accept in phabricator - https://phabricator.wikimedia.org/T131622#2623491 (10mmodell) 05Open>03Resolved a:03mmodell Ok I've enabled it. [16:46:22] 06Release-Engineering-Team (Long-Lived-Branches), 03Scap3, 13Patch-For-Review: Create `scap swat` command to automate patch merging & testing during a swat deployment - https://phabricator.wikimedia.org/T142880#2623494 (10mmodell) [16:48:28] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 03releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#2623497 (10mmodell) [16:48:30] 06Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921#2623496 (10mmodell) [16:50:59] 05Gerrit-Migration, 10Differential, 10Phabricator, 13Patch-For-Review: Enable differential.allow-self-accept in phabricator - https://phabricator.wikimedia.org/T131622#2623503 (10Paladox) Thanks, I guess we should merge https://gerrit.wikimedia.org/r/281071 also :) [16:58:53] phuedx: my API-fu is not strong [16:59:13] That is what I managed to do so far, still working on it [16:59:44] Both Jenkins job and mediawiki_api gem need to be updated [17:02:10] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is OK: OK: Less than 100.00% above the threshold [0.0] [17:48:46] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2623702 (10greg) [17:56:01] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T144644#2623781 (10greg) [17:56:24] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2623785 (10greg) 05Open>03Resolved [17:56:43] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T142855#2623787 (10greg) a:03demon [17:56:53] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T143328#2623791 (10greg) a:03demon [17:57:15] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T144644#2606121 (10greg) a:03thcipriani [18:18:06] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10DBA, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2623856 (10dduvall) [18:18:11] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10DBA, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2409864 (10dduvall) We're on for 9/15 @ 1500-1700 UTC and I've sent... [18:43:41] 10Beta-Cluster-Infrastructure, 10Parsoid, 10VisualEditor, 10VisualEditor-MediaWiki, and 2 others: On beta cluster, images are replaced with failed (404) links to Special:FilePath when you load VE - https://phabricator.wikimedia.org/T144884#2623953 (10Jdforrester-WMF) a:03AlexMonk-WMF [18:44:18] 10Beta-Cluster-Infrastructure, 10VisualEditor: Several nodes are not being rendered on Edit Mode in VE in beta - https://phabricator.wikimedia.org/T144883#2623961 (10Jdforrester-WMF) a:03AlexMonk-WMF [19:35:42] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:06:22] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2624342 (10Aklapper) 05Open>03stalled @Mbrt: Can you describe what you want to test out that would need a dedicated wiki project to be added? [20:15:44] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:36:52] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:41:57] Yippee, build fixed! [20:41:57] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #143: 09FIXED in 56 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/143/ [20:42:07] Yippee, build fixed! [20:42:07] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #143: 09FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/143/ [20:53:52] !log testing scap 3.2.5-1 on beta cluster [20:53:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:54:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:57:14] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:57:50] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:58:18] ^ fixing [20:58:23] ostriches i finally got polygerrit setup, lol they need to improve it alot, since i have to redirect most of the links to /r/ [20:58:30] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:08:45] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2622314 (10hashar) As @legoktm said, the Jenkins job will run on a fresh VM with no leftover from a previous build and then just `tox`. For CI you can clone integration/config and in zuul/layout.yaml and... [21:09:15] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [21:11:53] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:12:13] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:12:49] RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:32] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:04] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:16:06] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2624596 (10hashar) Looked at https://gerrit.wikimedia.org/r/#/c/308944/3 From what I can tell, the script goes through various source repositories, retrieve them with their appropriate source management... [21:18:34] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:22:08] ^ also fixing [21:23:49] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:43:38] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2624626 (10hashar) > Does the test suite needs all of them or only a subset, if we can narrow down the list, it would be quite nice. I have ran the test suite and it does not rely on any package for now.... [21:53:34] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [21:58:49] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:34] 06Release-Engineering-Team, 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2624661 (10greg) >>! In T144661#2618330, @jcrespo wrote: >> It's kind of... [22:17:54] 10Beta-Cluster-Infrastructure, 10VisualEditor, 07Verified: Several nodes are not being rendered on Edit Mode in VE in beta - https://phabricator.wikimedia.org/T144883#2624664 (10Tanveer07) [22:20:31] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:39:58] 06Release-Engineering-Team, 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2624715 (10bd808) >>! In T144661#2624661, @greg wrote: > So, here's a pr... [22:44:08] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2624723 (10dachary) @hashar thanks for your review. Indeed the additional packages are not immediately needed since I'm still to create the tests that need them. I'm asking because they need to be there b... [22:50:41] 10Continuous-Integration-Config: FLOSSbot continuous integration - https://phabricator.wikimedia.org/T145179#2624748 (10hashar) I did a bit more review on https://gerrit.wikimedia.org/r/#/c/308944/3 with suggestions. I will get a closer look next week at the various bin packages that are needed and probably jus... [22:52:07] fun friday code review for thcipriani, ostriches, twentyafterfour: https://gerrit.wikimedia.org/r/#/c/303923/ [22:52:33] looks fun so far [22:57:56] E_ITSFRIDAY [23:00:30] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:52] E_ITSFRIDAY == Merge all the things? [23:04:00] I mean, I guess I know it's supposed to be the file descriptor, but I think you need a timeout, too. [23:04:10] * thcipriani continues thoughts from reviews in irc [23:08:56] thcipriani: yeah. this is why we have code review :) [23:09:09] So I think I should just remove the --wait [23:09:25] and make it indefinitely blocking [23:09:45] I *think* that's right [23:10:01] "By default, if the lock cannot be immediately acquired, flock waits until the lock is available." [23:10:03] I'm mostly just playing with gnu parallel and trying to think about this :) [23:10:53] cool, yeah, that looks right to me [23:11:52] hmmm.. can you use a variable for a fd in bash? That could make the 9's make more sense [23:12:35] yuck, nope [23:14:56] there's probably some horrible exec that'll make it work [23:15:11] but that's probably not helping anything :) [23:15:31] yeah, you have to use exec or magic automatic fd allocation [23:15:49] and in this case I don't think magic allocation would work [23:16:19] * bd808 should have rewritten this script in python years ago [23:19:05] ApiRevThankIntegrationTest::testMediaWikiTestCaseParentSetupCalled BadMethodCallException: Call to a member function getId() on a non-object (null) [23:19:09] Anyone recognise this? [23:40:45] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: List all SWAT deployers for each window (no longer segment) - https://phabricator.wikimedia.org/T144297#2624792 (10greg) Currently the list of people is: 06:00 Pacific: Antoine (hashar), Sébastien (Dereckson), addshore, or Katie (aude) 11:00 Paci... [23:48:24] 10Deployment-Systems, 06Release-Engineering-Team: Require an associated task with each SWAT item - https://phabricator.wikimedia.org/T145255#2624798 (10greg) [23:50:22] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: Require an associated task with each SWAT item - https://phabricator.wikimedia.org/T145255#2624822 (10greg) [23:51:48] 10Deployment-Systems, 03Scap3, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#2624830 (10thcipriani) 05Resolved>03Open @fgiunchedi could I get you to upload scap_3.2.5-1 to carbon? New release should fix up {T145194}, {T144244}, and add a few other backwa... [23:51:59] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: Require an associated task with each SWAT item - https://phabricator.wikimedia.org/T145255#2624798 (10Krenair) Simple config cleanups need a task? [23:55:07] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: Require an associated task with each SWAT item - https://phabricator.wikimedia.org/T145255#2624840 (10greg) >>! In T145255#2624835, @Krenair wrote: > Simple config cleanups need a task? good point... that's one exception. [23:56:09] greg-g, to be honest I'm not convinced adding an extra rule there will help [23:56:16] it sounds like the concern is the existing rule is not being enforced