[00:01:28] I need to gain a user role on betalabs metawiki. Do I ask an admin, or is there an onwiki process? [00:01:36] question about beta labs, when i run `mwscript eval.php enwiki` this runs it as the apache user, but 'tmp/mw-UIDGenerator-UID-88' which is used by core UIDGenerator is owned by 'mwdeploy' [00:01:57] so question 1) does the web run as mwdeploy or apache? probably apache? [00:02:10] and question 2) if it runs as apache, what ran code as mwdeploy instead? [00:02:42] (this is on deployment-bastion) [00:02:58] <^d> (1) should be apache. [00:03:09] <^d> If it's not...that seems wrong. [00:03:32] <^d> No idea on (2) offhand. [00:03:59] yea, i just poked at deployment-apache0[12] and both have /tmp/mw-UIDGenerator-UID-88 owned by apache, so something on deployment-bastion is running as mwdeploy when it should be apache [00:04:14] probably a beta only problem though, i'll just file a bug [00:04:39] i can just run my mwscript tests from -apache01 [00:05:02] <^d> awight: Just find a bcrat I guess. I doubt we need more than that. [00:05:04] <^d> It's beta! [00:05:19] ^d: there are... so few bureaucrats tho... [00:05:21] theres a maint script for that ;) [00:05:33] <^d> I shall make myself a bcrat. [00:05:56] maybe I can do that too [00:06:10] I gave my self global developer or whatever it was, IIRC [00:06:58] yeah, global developer. allows you to set rights anywhere [00:07:23] * ^d gave himself --sysop and --bureaucrat on metawiki [00:07:50] ^d: well sorry for the one-off request: I need "Translator administrator" for user "Awight (WMF)" [00:07:58] urp "Translation administrator" [00:08:00] <^d> Also, somebody broke wmf-config. [00:08:07] that could have been me. [00:08:17] ^d: can u link to the breakage? [00:08:25] <^d> Grrr "You do not have permission to add or remove user rights." [00:08:35] blargh. [00:08:43] <^d> It's on the command line [00:08:45] <^d> Warning: array_key_exists() expects parameter 2 to be array, integer given in /mnt/srv/scap-stage-dir/php-master/includes/SiteConfiguration.php on line 244 [00:08:45] <^d> PHP Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/scap-stage-dir/php-master/includes/SiteConfiguration.php on line 305 [00:08:45] <^d> Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/scap-stage-dir/php-master/includes/SiteConfiguration.php on line 305 [00:08:52] We're talking about http://deployment.wikimedia.beta.wmflabs.org right awight? [00:08:58] Krenair: yeah [00:09:14] "Awight (WMF)" does not exist [00:09:39] ebernhardson: scap on deployment-bastion runs as mwdeploy. It may have done something wrongly with sudo [00:09:50] Krenair: I don't know how the machines are related, but the wiki I need perms on is http://meta.wikimedia.beta.wmflabs.org [00:09:55] Or http://meta.wikimedia.beta.wmflabs.org/ ? [00:09:55] ahh [00:10:09] <^d> Oh dur, I gave myself on meta. [00:10:18] Krenair: I'm looking at the deployment wiki for how to add myself to the admins [00:10:30] ^d: ha, sorry again. enter the labyrinth! [00:11:18] awight, done [00:11:32] Krenair: rad, thank you [00:12:01] that did it! [00:12:25] User rights changes are *supposed* to be handled from deployment [00:12:28] But um [00:12:40] Well that doesn't work for groups which aren't configured on deployment [00:13:03] cough https://gerrit.wikimedia.org/r/#/c/36330/ cough [00:13:57] Krenair: wow, you have much sticktoitiveness [00:54:19] 3Wikimedia Labs / 3tools: no recent dumps in /public/dumps/public/ - 10https://bugzilla.wikimedia.org/67909 (10bgwhite) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier No recent dumps are going into /public/dumps/public/ Appears the last one came in around June 25th. [01:22:02] Hi. I'm trying to work flask in tools-lab. I did the following steps: https://wikitech.wikimedia.org/wiki/Setting_up_Flask_cgi_app_as_a_tool but I get a 404 error. What can be the problem? :-( [01:28:59] when I run python cgi.py, I get KeyError: 'SERVER_NAME' A server error occurred. Please contact the administrator. [01:42:08] rohit-dua: i know nothing about python or flask, but based on general web server knowledge, SERVER_NAME is typically an environment variable set by the server before calling a cgi script [01:42:24] rohit-dua: so your cgi.py is expecting to be called by a webserver with all the different environemnt variables a web server would set [01:44:05] perhaps there is a different entry point for cli usage than cgi.py ? [01:48:37] For setting up flask, the "virtualenv python" should be run in ~/ or public_html/ [01:52:50] Coren: In case you happen to be online still: tools-exec-11, -12, and -13 can't send mail by piping to /usr/sbin/exim -odf -i user@example.com. Permission denied on writing to /var/log/exim4/mainlog. [01:53:03] (unless you tell me otherwise, I'll file a bug in a little bit) [02:06:04] 3Wikimedia Labs / 3tools: Can't send email from tools-exec-11, -12, or -13 - 10https://bugzilla.wikimedia.org/67912 (10Brad Jorsch) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I have a daily cronjob that pipes a status report to /usr/sbin/exim -v -odf -i. This evening, I noticed that the job was su... [02:06:20] * anomie files bug 67912 [02:45:03] 3Wikimedia Labs / 3tools: no recent dumps in /public/dumps/public/ - 10https://bugzilla.wikimedia.org/67909#c1 (10Marc A. Pelletier) 5UNCO>3RESO/DUP Cause by out of space; ticket (conceptually) duplicates the older one where status info is given. *** This bug has been marked as a duplicate of bug 48894... [02:45:03] 3Wikimedia Labs / 3tools: Include pagecounts dumps in datasets - 10https://bugzilla.wikimedia.org/48894#c23 (10Marc A. Pelletier) *** Bug 67909 has been marked as a duplicate of this bug. *** [07:39:32] Is there a file size limit for /home ? I got a file mysteriously truncated affter 400 MB exactly [09:03:17] 3Wikimedia Labs / 3tools: Performance problem on database server s5 using commonswiki - 10https://bugzilla.wikimedia.org/67602#c3 (10merl) I just divided another complex query into multiple queries for a performance test. This searches for included not existing images of a single article (there is only one i... [11:17:06] Nemo_bis: There isn't (or shouldn't be). [11:23:16] ok [11:28:47] scfc_de: hmm, https://bugzilla.wikimedia.org/show_bug.cgi?id=67912 [11:28:59] should we fix it by hand or investigate root cause? [11:42:55] YuviPanda|zzz: Let's see (running "pdsh -f 1 -g tools sudo ls -l /var/log/exim4/mainlog"). [11:43:02] ok [11:47:47] And the main difference is that the file doesn't exist on -1[1-3] :-). [11:48:09] And /var/log/exim4 neither. [11:54:56] scfc_de: hmm, exim4 doesn't create it? [11:54:59] the package? [11:55:00] that's stupid [11:55:07] Created /var/log/exim4 with perm 2750 and owner Debian-exim:adm, and anomie's test now works. *But*: I also received the mail from my previous test, so the directory not existing doesn't seem to stop exim4 from sending mail. I suggest we (I) create the directory on -12 and -13 as well and then file a bug to track why it wasn't created in the first place. [11:55:50] right [11:58:33] !log tools tools-exec-11, tools-exec-12, tools-exec-13: mkdir -m 2750 /var/log/exim4 && chown Debian-exim:adm /var/log/exim4; I'll file a bug why the directory wasn't created later [11:58:37] Logged the message, Master [11:59:28] Respectively, we could use bug #67912 for that ... [12:34:19] 3Wikimedia Labs / 3tools: Can't send email from tools-exec-11, -12, or -13 - 10https://bugzilla.wikimedia.org/67912#c1 (10Tim Landscheidt) a:5Marc A. Pelletier>3Tim Landscheidt My testing showed that mails do get through; but I created /var/log/exim4 with 2750 and Debian-exim:adm anyhow on -11, -12 and -... [12:39:10] YuviPanda|zzz: Did you receive the mails from anacron about /var/log/account not existing as well? This looks like the problem isn't limited to exim4, but that more packages did not do what they should do on installation. [12:44:56] I think it's related to biglogs. [12:48:29] And indeed, if I look at the shadowed /var/log, there all the directories are. Fuck. [12:55:32] 3Wikimedia Labs / 3tools: Can't send email from tools-exec-11, -12, or -13 - 10https://bugzilla.wikimedia.org/67912#c2 (10Tim Landscheidt) Okay, the culprit has been found: role::labs::lvm::biglogs hid the "original" /var/log where inter alia exim4 had created its directory. This causes several other packag... [12:58:19] !log tools tools-exec-11, tools-exec-13: Disabled queues in preparation of reboot [12:58:21] Logged the message, Master [13:00:07] !log tools tools-exec-11, tools-exec-13: qmod -r continuous@tools-exec-1[13].eqiad.wmflabs in preparation of reboot [13:00:09] Logged the message, Master [13:31:32] 3Wikimedia Labs / 3tools: Performance problem on database server s5 using commonswiki - 10https://bugzilla.wikimedia.org/67602#c4 (10Marc A. Pelletier) 5NEW>3ASSI Ah, there's clearly something stupid in the way the query is planned, because: select * FROM commonswiki_f_p.image where img_name in('Icon_to... [14:34:14] scfc_de: yeah [14:34:30] scfc_de: bah, looks like applying biglogs on staartup alone won't fix this [14:41:29] YuviPanda: I think (or rather fear :-)) that the LVM stuff can't be managed automagically in Puppet at all and we'll indeed need to move it to wikitech/OpenStack or do it manually altogether, at least for volumes apart from maybe /mnt or something else uncontroversial. [14:41:43] scfc_de: yeah, I think the images need to be redone [14:42:21] No, the images (if you mean the VM images by that) are fine. I mean the Puppet classes role::lvm::biglogs & Co. [14:42:34] scfc_de: wouldn't putting them in the images solve the problem for everyone? [14:42:45] or can LVM stuff not be put in the images? [14:43:56] Ah, you mean on instance creation create an /var/log LVM volume and reference that on initial boot in the image so that no logs get written to the "wrong" volume? Hmmm. Sounds complicated. [14:44:44] scfc_de: right. or just don't use lvm for /var. There's already a /var partition, only problem is that it's 2G [14:44:55] so increase that size, enlarge root to also take up /tmp [14:45:14] fundamentally, don't use lvm for /var at all :) [14:47:51] That would probably a quick solution that could last for quite some time :-). Unless we are short on disk space, that would save us a lot of headache. [14:48:36] scfc_de: yeah, and I don't think we're short on disk space :) [14:53:25] If I'm not missing something, in Tools even on the biglogs hosts /var/log doesn't use more than 2 GBytes, so we wouldn't have to increase /var from 2 to 8, 4 or 6 would probably be enough for our use cases. [14:54:49] yeah [14:55:32] 3Wikimedia Labs: Dumps of Wikidata not available - 10https://bugzilla.wikimedia.org/67922#c1 (10Andre Klapper) 5NEW>3RESO/DUP *** This bug has been marked as a duplicate of bug 66362 *** [14:55:32] 3Wikimedia Labs / 3Infrastructure: latest dump not available again - 10https://bugzilla.wikimedia.org/66362#c10 (10Andre Klapper) *** Bug 67922 has been marked as a duplicate of this bug. *** [15:37:32] somebody here who can tell me whats wrong with my tools.wmflabs.org/tmg/ ????? [15:39:07] http://tools.wmflabs.org/tmg/test.html works. [15:39:40] http://tools.wmflabs.org/tmg/index.php does nothing but outputting html. no database. whats blocked there and why? [15:41:40] Thiemo_WMDE: Hmmm. If I "become tmg" and then "cd public_html && php index.php", the output comes immediately. [15:43:46] Why "curl -H 'Host: tools.wmflabs.org' -I http://tools-webgrid-01:4178/tmg/" => "curl: (7) couldn't connect to host"? [15:44:06] So nothing listening? [15:45:39] Ouch, -02. [15:47:28] Thiemo_WMDE: Did you restart the webservice? [15:47:43] yes. still no idea what went wrong. [15:53:49] Why did I get a new DB user/pass? [15:54:11] Dispenser: What do you mean? [15:54:37] I was u2815 this morning now I'm s51507 [15:55:48] are you sure you didn't switch from user to tool account? [15:56:14] Yes, it just happened like 5 mins ago [15:57:21] your right, wonder how that happened [15:57:33] Dispenser: gifti is right. Your ~dispenser/replica.my.cnf still says u2815, your ~tools.dispenser/replica.my.cnf s51507. [15:59:34] too many screens and accounts make things confusing [16:05:31] Oh, that's easy to solve: Just don't name a tool after a person. [16:08:20] 3Wikimedia Labs: Replace role::labs::lvm::biglogs with increasing the default /var partition - 10https://bugzilla.wikimedia.org/67926 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None Bug #67912 showed that role::labs::lvm::biglogs is fundamentally broken: You can't move around/shadow /var/log in a... [16:13:40] anyone know anything about security certificates on labs? or have a link to info on it on wikitech? [16:13:50] firefox is blocking my gadget b/c of mixed active content [16:13:53] :( [16:14:09] basically their help page says "get a security certificate" so [16:17:20] haha. also i just created a proxy with an invalid dns hostname and i can't deleted it because it's so malformed :) [16:19:29] mvolz: [15:53] [16:19:29] Dispenser: What do you mean? [15:54] [16:19:29] I was u2815 this morning now I'm s51507 [16:19:32] are you sure you didn't switch from user to tool account? [15:55] [16:19:36] Yes, it just happened like 5 mins ago [15:56] [16:19:39] your right, wonder how that happened [15:57] [16:19:42] Dispenser: gifti is right. Your ~dispenser/replica.my.cnf still [16:19:45] says u2815, your ~tools.dispenser/replica.my.cnf s51507. [16:19:48] too many screens and accounts make things confusing [15:59] [16:19:51] Oh, that's easy to solve: Just don't name a tool after a person. [16:19:55] [16:05] [16:19:55] *** YuviPanda (~yuvipanda@mediawiki/yuvipanda) is now known as YuviPanda|zzz [16:19:58] [16:07] [16:20:02] Wikimedia Labs: Replace role::labs::lvm::biglogs with increasing [16:20:05] the default /var partition - https://bugzilla.wikimedia.org/67926 [16:20:08] (Tim Landscheidt) NEW p:Unprio s:normal a:None Bug #67912 showed [16:20:11] that role::labs::lvm::biglogs is fundamentally broken: You can't [16:20:15] move around/shadow /var/log in a... [16:08] [16:20:18] anyone know anything about security certificates on labs? or have a [16:20:21] link to info on it on wikitech? [16:13] [16:20:24] firefox is blocking my gadget b/c of mixed active content [16:20:27] :( [16:20:29] basically their help page says "get a security certificate" so [16:14] [16:20:32] haha. also i just created a proxy with an invalid dns hostname and i [16:20:35] can't deleted it because it's so malformed :) [16:17] [16:20:38] ERC> mvolz: https://developer.mozilla.org/en-US/docs/Security/MixedContent suggests you just need to replace an inclusion of http:// with https://. Regarding the latter, could you file a bug for that, please? [16:20:41] scfc_de: problem with copy&paste? [16:20:41] Ooops ... Sorry. [16:20:52] Dispenser: Only with paste. [16:22:00] scfc_de: thanks will do [16:27:23] scfc_de: that did the trick. phew! [16:27:27] tnx [16:28:20] scfc_de: btw, https://github.com/BrightcoveOS/Diamond/blob/master/src/collectors/gridengine/gridengine.py has the current upstream gridengine collector [16:29:00] scfc_de: am I filing a bug report in order to get an admin to delete the malformed entry, a bug report that malformed entries are being added but can't be deleted, or both? [16:29:06] scfc_de: wonder if that'll be 'enough' for us or if we need to add more [16:29:11] I really should read up on grid engine soon [16:32:05] mvolz: I think you can fold them into one. [16:34:16] YuviPanda: Hard to see what's actually been collected :-). I think at the moment (modulo Ganglia not being available) we have jobs running, waiting and in error. Load and memory ... do we handle this in a general by-host stat because these need to be monitored by all? [16:39:19] 3Wikimedia Labs / 3tools: Can't delete NovaProxy instance with malformed DNS hostname - 10https://bugzilla.wikimedia.org/67927 (10marielle.volz) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier The entry that needs to be deleted is in the project "services" and it will be obvious which one it is. :) W... [17:53:03] !log tools tools-exec-11: Moved log files around, rebooted, restored iptables and reenabled queue ("qmod -e {continuous,task}@tools-exec-11...") [17:53:05] Logged the message, Master [17:53:40] (But why is there an Apache running on -11?!) [17:57:51] !log tools tools-exec-11: Stopping apache2 service; no clue how it got there [17:57:53] Logged the message, Master [18:13:18] Coren: tools.wmflabs.org fingerprint changed? [18:15:46] liangent: Not in ages; but I'm pretty sure you want tools-login not tools [18:16:00] Coren: right thanks [18:21:02] 3Wikimedia Labs: Replace role::labs::lvm::biglogs with increasing the default /var partition - 10https://bugzilla.wikimedia.org/67926#c1 (10Marc A. Pelletier) This is definitely the wrong solution; doing it this /at best/ simply delays the problem and fixes absolutely nothing. Anything that stuff things into... [18:21:47] 3Wikimedia Labs / 3tools: Can't send email from tools-exec-11, -12, or -13 - 10https://bugzilla.wikimedia.org/67912#c3 (10Tim Landscheidt) -11 is done, on -13 runs a job (2281968) that based on past runs could take up to another 12 hours to complete. [19:42:05] 3Wikimedia Labs / 3deployment-prep (beta): Beta should not use productions interwiki.cdb - 10https://bugzilla.wikimedia.org/67931 (10Marius Hoch) 3NEW p:3Unprio s:3normal a:3None Beta should overwrite $wgInterwikiCache to point to a labs specific cdb. That might break some stuff on beta relying on t... [19:43:20] Do we have more external IPs? [20:44:45] http://redmine.lighttpd.net/issues/814 Allow setting a default encoding type WONTFIX. Feels like WMF, can't make shit easy. The wiki needs a section, "How to Make our the Webserver Work Non-Sporadically" [21:05:24] scfc_de: still around? [21:05:47] I was thinking of the webservice problem, wondering if a watchdog running on an arbit host that's a submit host can actually monitor and restart as appropriate [21:06:23] Mother FU*KR: Duplicate config variable in conditional 0 global: mimetype.assign [21:07:49] scfc_de: does qstat give out info on why a service was killed? [21:07:52] let me read the man page [21:21:36] YuviPanda: No, I don't think so. My concept was: Pass job number to watchdog; watchdog checks if job's still there; if so, sleep 1 minute and reset "$escalator"; otherwise, start webservice again and note job number, mail maintainers, sleep $escalator minutes and $escalator *= 2. [21:22:54] One thing I also noted was that at the moment we name webservices differently depending on lighttpd vs. Tomcat. I'd like to merge them to one webservice-$TOOL, as each tool can only have one webservice :-). [21:23:30] *Pass job number = Pass job number + whatever options the user specified [21:24:21] scfc_de: right [21:24:30] scfc_de: but then it'll have to have rights to start the jo [21:24:31] b [21:28:50] The watchdog job would run as the tool (in the same way as the webservice), so all it has to do is "webservice start $OPTIONS" if the host is a submit host. [21:30:14] hmm, right [21:34:24] I shall sleep now [21:34:25] night [21:38:15] Good night!