[00:08:28] bd808: still complaining :( [00:09:27] * bd808 tries [00:09:57] cd /srv/mediawiki-vagrant/ && vagrant up --provision [00:10:06] it's trying to boot now... [00:12:48] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Brclz was created, changed by Brclz link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Brclz edit summary: Created page with "{{Tools Access Request |Justification=Make [https://github.com/ValentinBrclz/WP-OubliSignature-Bot this cutie] - written in Node.JS that sign when new users forget - work. |Co..." [00:13:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Brclz was modified, changed by Brclz link https://wikitech.wikimedia.org/w/index.php?diff=218128 edit summary: Seems to be better [00:14:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Brclz was modified, changed by Brclz link https://wikitech.wikimedia.org/w/index.php?diff=218131 edit summary: [00:15:46] YuviPanda: quarry.wmflabs , is it behind "dynamicproxy"? [00:15:53] mutante: yes [00:16:06] mutante: dynamicproxy doesn't have a httpsonly option unfortunately [00:16:14] YuviPanda: do we have http-https enforcement for anything [00:16:16] ah [00:16:37] ok, but it's the right module i am looking at then [00:16:49] module/dynamicproxy [00:17:01] mutante: yes [00:17:03] also had the word "instanceproxy" or something on my mond [00:17:04] mind [00:17:06] ok, thanks [00:17:07] mutante: domainproxy.conf [00:17:13] mutante: yeah, instanceproxy was killed a while ago [00:17:17] gotcha [00:17:46] domainproxy.lua ? [00:18:39] and .conf [00:19:41] jdlrobson: the instance is still provisioning but it seems to be working just fine. It got to the part where it cloning mw/core which is sloooooow [00:20:12] bd808: any ideas why that is not working for me? :( [00:21:14] jdlrobson: what does `type -a vagrant` say for your shell? [00:21:28] vagrant is /usr/bin/vagrant [00:21:35] bam! [00:22:09] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1854539 (10Dzahn) it's behind dynamicproxy (module dynamicproxy) and that does not have an option for https-only yet. we need to add that in general in dynamicproxy for all projects using... [00:22:20] You need to log out and log back in after applying the role to pick up the alias that makes vagrant == /usr/local/bin/mw-vagrant [00:22:45] (it's in the doc but maybe not prominent enough) [00:23:52] jdlrobson: you can do `source /etc/profile.d/alias-vagrant.sh` to fix your current session too [00:25:29] now just have to work out what's going on with http://future-wikipedia.wmflabs.org/ :) [00:34:22] jdlrobson: did you point the proxy at port 8080 on the host? [00:34:58] I did on http://futurewiki.wmflabs.org/ [00:36:28] 6Labs, 10Labs-Infrastructure, 6operations: add a https-only option to dynamicproxy - https://phabricator.wikimedia.org/T120486#1854628 (10Dzahn) 3NEW [00:36:48] 6Labs, 10Labs-Infrastructure, 6operations, 7HTTPS: add a https-only option to dynamicproxy - https://phabricator.wikimedia.org/T120486#1854641 (10Dzahn) [00:37:09] 6Labs, 10Labs-Infrastructure, 6operations, 7HTTPS: add a https-only option to dynamicproxy - https://phabricator.wikimedia.org/T120486#1854628 (10Dzahn) [00:37:12] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1499757 (10Dzahn) [00:37:39] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1854656 (10Dzahn) a:5Dzahn>3None [00:48:32] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review: labcontrol1001 and 1002 running web servers on 80 and 443 for no reason - https://phabricator.wikimedia.org/T120449#1854772 (10Dzahn) Maybe we can add base::firewall and ferm rules to limit the access to what needs to talk to it and just not open holes for the... [00:48:58] sigh labs hates me... i think it's time to call it a weekend [01:24:06] 10PAWS, 7Documentation, 7Pywikibot-documentation: Expose the fact that PWB is available more - https://phabricator.wikimedia.org/T120072#1854913 (10yuvipanda) MOTD added! https://github.com/yuvipanda/paws/pull/4 can tweak over time. [07:09:23] Hi [07:10:20] I have some question about Grid [08:29:14] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org, and 4 others: MWException after account creation on wikitech - https://phabricator.wikimedia.org/T117553#1855216 (10Legoktm) Thanks @ori, I confirmed this is fixed by creating User:BugT117553 on wikitech. [09:42:36] Wheatley_: ? [10:17:13] i insert commend "jsub -once -cwd python pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" -always" [10:17:24] but it doesn`t worked [10:27:54] Wheatley_: I have found you need to raise the memory-limit when running pywikibot-scripts. [10:28:27] Wheatley_: last time, I ran with -mem 2048k [10:28:37] Wheatley_: maybe an exaggeration, but it worked. [10:43:57] Wheatley_: any error messages? [10:44:28] you might want to supply a job name btw [11:06:40] /usr/bin/python2.7: can't open file 'pwb.py': [Errno 2] No such file or directory [11:24:20] oh that [11:24:43] you need to supply the full path of pwb.py [11:26:11] like "python /shared/pywikipedia/core/pwb.py" or "python ~/pywikibot/pwb.py" [11:26:27] good day [11:26:58] how do I know if jobs on my crontab are being submited? [11:27:16] I tried [11:27:18] ? [11:27:18] 11 20 * * * jsub python bot.py [11:27:27] but no job was submited to the grid [11:28:11] oh I think I know why [11:31:19] minutes come before hours in the crontab syntax [11:36:28] it still not worked [11:36:35] jsub -once -cwd python /shared/pywikipedia/core/pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" -always -mem 2048k [11:40:23] Is there any way we can restart Misza13 Importer bot? [11:40:37] https://en.wikipedia.org/wiki/User_talk:Misza13#Wikinews_Importer_Bot [11:49:59] Wheatley_: what's the error message this time? [11:51:31] lbertolotti: well, if Misza13 or another maintainer/operator restarts the bot, yeah [11:52:36] zhuyifei1999_: what do u mean "another operator"? [11:53:14] some bots are operated by multiple users [11:54:08] zhuyifei1999_: but this doesn't seem the case [11:54:34] true http://tools.wmflabs.org/?tool=wikinews-importer [11:54:49] you might want to fork it [11:55:20] ? [11:56:05] like copy the code, get bot approval and then run the new importer bot [11:57:11] or otherwise, wait for Misza13. I don't think tool labs admins will restart it [11:58:38] I see [12:05:37] it did not show any message [12:06:08] can you see my account leedors@tools-bastion ? [12:07:12] ? [12:08:29] "-mem" 2048k goes before "python", and I have no idea what "-cwd" is for [12:09:18] nvm, I see [12:12:33] * "-mem 2048k" [12:47:23] jsub -once -cwd python /shared/pywikipedia/core/pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" -always "-mem 2048k" [12:47:36] it doesn`t work [12:51:32] some things might be related: [12:52:00] have you tried using a tool account (service account)? [12:52:14] on cmd? [12:52:16] yes [12:52:18] also "-mem 2048k" go before python [12:52:50] that's "jsub -once -mem 2048k -cwd ........" [12:53:07] what do you mean by "on cmd?" [12:55:35] jsub -once -cwd -mem 2048k python /shared/pywikipedia/core/pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" -always [12:55:49] it is not working [12:55:59] error log? [12:56:27] usr/bin/python2.7: error while loading shared libraries: libssl.so.1.0.0: failed to map segment from shared object: Cannot allocate memory [12:56:50] ugh [12:57:27] seems like memory used up [12:57:44] 2048k = 2m, I just noticed [12:58:02] try -mem 2g [12:58:12] or -mem 1g [12:58:49] Wheatley_: yeah, my mistake.. that was supposed to be -mem 2048m [12:58:57] not 2048k :) [13:01:24] lol I just read jsub code and found the default is 256M [13:03:10] zhuyifei1999_: yup... and that's a bit too low IMO [13:04:16] 2MB... You are trying to squeeze that into an Amiga? XD [13:05:01] Stigmj: true, how about filing a phab ticket to double the default? [13:06:39] most scripts needs to have a higher mem, and those oom messages can confuse people [13:07:22] eg. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Why_am_I_getting_errors_about_libgcc_s.so.1_must_be_installed_for_pthread_cancel_to_work.3F [13:14:02] 6Labs, 10Tool-Labs: Raise the default memory allocation to jsub/jstart jobs from 256M to at least 400M. - https://phabricator.wikimedia.org/T120517#1855399 (10Stigmj) 3NEW [13:14:16] zhuyifei1999_: done.. [13:16:40] I'll get a patch, but this grid stuff will be gone in a few months I think [13:17:06] (replaced by k8s) [13:21:13] it`s not working [13:21:22] when execute commend [13:21:28] it`s on qstat [13:21:36] but disapper soon [13:21:42] something weird going on. appearantly jsub isn't managed by puppet [13:22:00] Wheatley_: come on, what's inside the error log? [13:22:07] nothing [13:22:12] ugh [13:22:36] what happens if you run python /shared/pywikipedia/core/pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" -always on bastion? [13:22:56] Your job 32378 ("python2") has been submitted [13:23:03] qstat -> nothing [13:23:15] disappered soon [13:23:18] I mean without the jsub part [13:23:27] ok [13:23:50] nothing [13:23:51] I think bot-testing on bastion is fine [13:23:55] ugh [13:24:07] wait [13:25:59] I don't see the error [13:26:27] https://www.irccloud.com/pastebin/F88DHgkl/ [13:27:25] it`s ok when execute as "pwb.py template -family:wiktionary "한국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" at outground(?) [13:27:42] outground? [13:27:46] foreground [13:27:50] cmd [13:28:08] only for testing [13:28:21] (temporary testing) [13:29:26] (I think) [13:46:32] (03PS1) 10Zhuyifei1999: jsub: Raise default memory allocation to 400M [labs/toollabs] - 10https://gerrit.wikimedia.org/r/257062 (https://phabricator.wikimedia.org/T120517) [14:05:05] it not worked [14:05:16] python pwb.py template -family:wiktionary "한 국어" -subst -cc "-summary:위키낱말사전 틀 체계를 걷어내기 위함" [14:05:18] not work [14:16:14] Wheatley_: what is not work? [14:16:25] error messages, etc? [14:16:56] not error messages [14:17:02] just nothing [14:18:11] ugh [14:25:23] maybe an admin can investigate on your configs to see what's wrong [14:28:33] ok [14:53:03] 6Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#1855528 (10Nemo_bis) > Replaced by a 301 redirect to https://www.mediawiki.org/wiki/Toolserver:Main_Page. Thanks for going the extra mile with a redirect! [16:19:17] 6Labs, 10Tool-Labs: Redirect //stable.toolserver.org/geohack/geohack.php requests - https://phabricator.wikimedia.org/T120526#1855664 (10Nemo_bis) 3NEW [16:19:48] 6Labs, 10Tool-Labs: Redirect //stable.toolserver.org/geohack/geohack.php requests - https://phabricator.wikimedia.org/T120526#1855672 (10Nemo_bis) [21:31:33] YuviPanda: What is the status of nagf? [21:31:41] It seems to be very slow in http response [21:31:58] looking at the account on tools-login shows the account has been domant for the past 2 months [21:32:02] suggesting it runs elsewhere [21:32:07] did you put it in k8s already? [21:33:06] is it slow? [21:33:15] yes it's been on k8s for a while now [21:33:24] it comes and goes [21:33:28] how do I inspect it? [21:33:33] How does it run in k8s? [21:33:47] Is there a k8s.manifest or some such? [21:33:53] or a k8sstat? [21:34:06] Krinkle: hmm, the main page loads fast for me but fetching anything is slow, I wonder if that's just wikitech being slow [21:34:19] Krinkle: so there's kubectl which does all the things [21:34:33] wikitech things should be cached [21:34:39] (and are) [21:34:42] anyway [21:34:47] ah indeed [21:34:53] let me know how I can inspect it, and how I can break it [21:34:54] second request to same project is fast [21:34:55] (or fix it) [21:35:20] Krinkle: so it's in the 'yuvipanda/nagf' docker container on dockerhub (we don't have our own registry yet) [21:35:32] Krinkle: if you create a dockerhub account I can give you push access to it [21:35:40] Hm.. [21:35:51] if you want to update it [21:35:54] so it's completely detached from tools-login? [21:35:55] (adding credentials now) [21:36:06] (that's okay, just checking) [21:36:07] no, looking at logs, exec, etc is from tools-k8s-bastion [21:36:17] but yeah, it's completely detached from all the gridengine stuff [21:36:32] sure [21:36:44] Krinkle: but be slightly warned - this is all super unpolished raw stuff. Eventually the experience would be similar to heroku, but we aren't there yet. [21:36:47] KrinkleMac$ ssh tools-k8s-bastion.eqiad.wmflabs [21:36:47] channel 0: open failed: administratively prohibited: open fail [21:37:05] bah [21:37:08] tools-k8s-bastion-01 [21:37:47] I'm okay being for testing. Just wanna make sure nagf doesn't go too deep into beta to the point where it's only testing k8s itself and none of our user infrastructure. [21:37:53] yeah, +1 [21:38:13] woo, nfs [21:38:18] I haven't seen these dotfiles in a while [21:38:59] What kind of commands would you run with kuberctl? [21:39:17] there is no become on this server [21:39:57] Krinkle: yeah, sudo su tools.nagf [21:40:10] Krinkle: so 'kubectl get pods' will get you the running pods [21:40:19] Krinkle: kubectl logs will show you the logs [21:40:35] Krinkle: kubectl exec -t -i bash will run a bash session inside the container so you can poke around stuff [21:41:03] Krinkle: to update code right now you have to push to yuvipanda/nagf on dockerhub and kubectl delete pod/ [21:41:26] the replication controller always ensures that at least one pod is running and that is latest (this is in the replication controller specification, which you can view with 'kubectl get rc -o yaml' [21:41:29] ) [21:41:31] so it'll create a new pod [21:41:33] 'kubectl get pods' as both myself ans tools.nagf produces an error [21:41:34] with the latest image [21:41:41] Krinkle: what error [21:41:44] error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused [21:41:56] on both tools-login and tools-k8s-bastion-0 [21:41:59] Krinkle: as nagf? [21:42:04] Yep [21:42:12] nagf on tools-k8s-bastion-01 [21:42:20] [21:42 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ [21:42:22] 127 $ kubectl get pods [21:42:24] NAME READY STATUS RESTARTS AGE [21:42:26] nagf-7hj5o 1/1 Running 0 8d [21:42:28] hmm [21:42:30] Krinkle: what's your PWD [21:42:32] err [21:42:34] CWD [21:43:34] Krinkle: credentials are in $HOME/.kube/config for tools.nagf [21:43:49] CWD shouldn't matter though [21:44:00] that error message is consistent with the config file not being picked up though [21:48:17] 10PAWS: Do not require users to type passwords into PAWS - https://phabricator.wikimedia.org/T120331#1856034 (10yuvipanda) a:3yuvipanda [21:50:51] YuviPanda: It works via sudo -iu instead of -su [21:50:57] probably one of the env vars is confusing it [21:51:04] $ kubectl cluster-info [21:51:04] Kubernetes master is running at https://tools-k8s-master-01.tools.eqiad.wmflabs:6443 [21:51:38] Krinkle: ah, interesitng. I just used sudo su...