[00:35:51] chrismcmahon: i broke beta again, fix in a minute [00:36:02] ori: ok [00:36:12] sorry, i'll have good news in a minute too [01:38:39] 3Wikimedia Labs / 3deployment-prep (beta): mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - 10https://bugzilla.wikimedia.org/65591#c10 (10Tim Landscheidt) Ah. And that seems to relate to the subversion module where users apparently were only allowed to execute "svnserve -... [03:14:36] ERROR 2006 (HY000): MySQL server has gone away [03:14:36] No connection. Trying to reconnect... [03:14:36] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [03:14:41] ERROR: Can't connect to the server [03:18:11] commonswiki, dewiki, wikidatawiki [03:19:23] 03:11 springle: filesystem issues on labsdb1002. stopped mysqld [03:25:40] ok, so labsdb1002 serves s2, s4 and s5 [04:40:59] <^d> Grrr. Is logging broken on beta? [06:36:10] 3Wikimedia Labs / 3tools: Install MLT and kdenlive on tool labs - 10https://bugzilla.wikimedia.org/69365#c4 (10prolineserver) Yes, that would be perfect. If it later turns out that the tool is heavily used and a separate project has some advantages I might have a closer look again. Thanks! [07:59:39] 3Tool Labs tools / 3wikibugs IRC bot: Wikibugs not showing changes: mail is not being delivered to toredis.py - 10https://bugzilla.wikimedia.org/70002#c1 (10Merlijn van Deen) 5NEW>3RESO/FIX Caused by a full partition on the SGE master server; fixed by Coren. [10:09:52] yuvipanda: UGH. ipython notebook 2.x broke ALL THE KEYBOARD SHORTCUTE [10:10:02] I hate people who break keyboard shortcuts [10:10:03] >:( [12:36:35] (03PS1) 10Krinkle: These are already going to #wikimedia-qa. Since all CI people are either in both or only in #wikimedia-qa, remove from here to reduce duplicate pings and overal noise. [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 [12:37:00] (03PS2) 10Krinkle: #wikimedia-dev: No longer report integration/* [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 [13:23:31] (03CR) 10Hashar: [C: 031] #wikimedia-dev: No longer report integration/* [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 (owner: 10Krinkle) [14:33:09] 3Wikimedia Labs / 3tools: dewiki / wikidatawiki replication (s5) has stopped on labsdb1001/labsdb1002 - 10https://bugzilla.wikimedia.org/69776#c2 (10Marc A. Pelletier) 5NEW>3RESO/WOR There is no current issue with either dewiki and wikidatawiki, and revisions are coming in with no significant lag (subsec... [14:40:24] 3Wikimedia Labs / 3tools: I can't access ptwiki_p database - 10https://bugzilla.wikimedia.org/69701#c3 (10Marc A. Pelletier) 5NEW>3RESO/FIX The maintenance script that maintains replica credentials was out of sync with the actual database configuration for a while; some users created during that interval... [14:41:24] 3Wikimedia Labs / 3tools: Some users can't connect to replica DB servers - 10https://bugzilla.wikimedia.org/69679#c1 (10Marc A. Pelletier) 5ASSI>3RESO/DUP *** This bug has been marked as a duplicate of bug 69701 *** [14:41:24] 3Wikimedia Labs / 3tools: I can't access ptwiki_p database - 10https://bugzilla.wikimedia.org/69701#c4 (10Marc A. Pelletier) *** Bug 69679 has been marked as a duplicate of this bug. *** [14:46:10] 3Tool Labs tools / 3[other]: Shared version of pywikibot on tools.wmflabs.org is broken - 10https://bugzilla.wikimedia.org/68215#c6 (10Marc A. Pelletier) a:5Marc A. Pelletier>3John Mark Vandenberg Assigning to jvdb; this appears to be fixed but I'll leave acknowledgement to him. [14:47:42] (03CR) 10Chad: [C: 031] #wikimedia-dev: No longer report integration/* [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 (owner: 10Krinkle) [15:21:08] 3Wikimedia Labs / 3tools: androidsdk::dependencies doesn't work on Trusty - 10https://bugzilla.wikimedia.org/69423#c2 (10Marc A. Pelletier) Please enumerate the dependencies you will actually need for a working SDK under Trusty. Probably one of the best method to do so with certainty is to create a Trusty i... [16:20:53] 3Wikimedia Labs / 3deployment-prep (beta): mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - 10https://bugzilla.wikimedia.org/65591#c11 (10Tim Landscheidt) (In reply to myself from comment #10) > [...] > So to migrate safely we would need to: > 1. Add the override to /usr/... [16:45:56] 3Wikimedia Labs / 3deployment-prep (beta): Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None per Antoine: Have labs ops raise our quota of cpu/mem/disk then create a... [16:49:54] 3Wikimedia Labs / 3Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049#c1 (10Antoine "hashar" Musso) Moving to 'infrastructure' component. Have to poke Andrew / Marc-André to figure out whether the quota for the... [16:53:53] 3Wikimedia Labs / 3Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049#c2 (10Andrew Bogott) What are your current quotas, and what do you need them raised to? It should be much of a problem in any case, overprovi... [16:54:24] 3Wikimedia Labs / 3Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049#c3 (10Andrew Bogott) Um... shouldn't. shouldn't be much of a problem. [17:05:31] chrismcmahon: how's beta looking today? [17:06:09] * yuvipanda waves at ori [17:06:21] yo yuvi [17:06:29] ori: still in Europe? [17:06:38] nope [17:06:43] ori: back in SF? [17:06:52] yep [17:07:04] ori: much better, thanks [17:45:24] 3Wikimedia Labs / 3tools: Default umask should be set to 0002 - 10https://bugzilla.wikimedia.org/46468#c6 (10Marc A. Pelletier) What is the nature of the issue that remains, if any? [17:46:53] 3Wikimedia Labs / 3tools: Unable to explain queries on replicated databases - 10https://bugzilla.wikimedia.org/48875#c17 (10Marc A. Pelletier) 5REOP>3RESO/WON Given the availability of the new tool, and the lack of desire upstream to change what they perceive as WAD, closing this. [17:47:42] 3Wikimedia Labs / 3tools: Define rules for using Ganglia for individual tool statistics - 10https://bugzilla.wikimedia.org/48737#c2 (10Marc A. Pelletier) 5NEW>3ASSI a:5Marc A. Pelletier>3Yuvi Panda This may become possible thanks to labmon1001; handing off to Yuvi. [17:51:17] 3Wikimedia Labs / 3tools: Show replication lags in Ganglia - 10https://bugzilla.wikimedia.org/48694 (10Marc A. Pelletier) 5REOP>3RESO/FIX [17:52:39] 3Wikimedia Labs / 3tools: Unable to explain queries on replicated databases - 10https://bugzilla.wikimedia.org/48875#c18 (10Yuvi Panda) I'll also note that MariaDB 10.1 has an 'ANALYZE' feature coming up that is open to everyone and that lets you evaluate queries without having to run them, so whenever it is... [17:57:08] 3Wikimedia Labs / 3tools: jstart doesn't signal out-of-memory kills to the user - 10https://bugzilla.wikimedia.org/50053#c1 (10Marc A. Pelletier) 5ASSI>3RESO/WON Users may request -m from jsub/jstart which are passed to qsub and behave as expected. I'd rather not increase the default amount of cron/grid... [17:58:24] 3Wikimedia Labs / 3tools: Replicate the Phabricator database to labsdb - 10https://bugzilla.wikimedia.org/50422 (10Marc A. Pelletier) [18:00:09] 3Wikimedia Labs / 3tools: Replicate the Phabricator database to labsdb - 10https://bugzilla.wikimedia.org/50422#c3 (10Yuvi Panda) We are also not switching Gerrit to Phabricator yet - just Bugzilla and RT. [18:01:32] Coren: You around? [18:01:39] a930913: What be up? [18:01:49] Coren: How was Wikimania? [18:02:17] a930913: hectic. :-) [18:02:49] Heh. Did Ed remember to say hello from me to you? [18:03:09] 3Wikimedia Labs / 3tools: Allow jvm non-cgi webapps - 10https://bugzilla.wikimedia.org/50453#c3 (10Marc A. Pelletier) 5NEW>3RESO/FIX That said, this is now a "normal" scenario and in use by some tools. The exact process is available: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Other_we... [18:03:37] a930913: ... probably. It's more likely that *I* forgot that he did. :-) [18:03:43] Coren: :D [18:03:54] Coren: re ^, does that mean portgranter now runs on all nodes? [18:04:48] yuvipanda: No; hence the pointer at the document that specifies one needs to use the (increasingly misname) webgrid-tomcat [18:04:49] Coren: Ok, holiday over :( Can you help me find out why my python cgis don't work unless in the cgi-bin directory? [18:04:58] heh [18:04:58] ok [18:06:05] a930913: ... I'm about the worse person to help with Python, ever. That said, it normally takes some explicit configuration and a bit of doing to use cgi-bin at all in the current scheme. Have you tried with the default config? [18:06:40] a930913: I usually poke valhallasw`cloud and legoktm for python on labs help [18:07:00] hi [18:07:04] * a930913 pokes valhallasw`cloud and legoktm again for good measure. [18:07:07] legoktm: o/ [18:07:18] tools.legobot@tools-login:~$ cat .lighttpd.conf [18:07:19] $HTTP["url"] =~ "^/legobot/cgi-bin" { [18:07:19] cgi.assign = ( "" => "" ) [18:07:19] } [18:07:26] that works for me [18:07:47] just change the path and see if it works? [18:08:39] I don't have that like that. [18:10:08] legoktm: less /data/project/cluestuff/.lighttpd.conf [18:10:56] oh, you're trying to use fastcgi [18:11:08] I just stole valhallasw`cloud's config for that [18:11:16] legoktm: Only for one project. [18:13:46] hm [18:13:49] I don't see why it won't work [18:13:56] you $ webservice restart 'd right? [18:14:06] legoktm: Yup. [18:24:31] hi ppl [18:24:39] is there a lab admin around ? [18:25:17] I've been asked by Ynhockey to help out with WLM [18:25:48] can I get access to the labs it runs on ? [18:26:31] also - it seems he also does not have access to it [18:28:03] ori: do you anticipate that hhvm on beta labs is reasonably stable at this point? [18:28:45] should be, yeah [18:31:54] ori: great. if you do find yourself using a debug build or something for any length of time, let me know? and btw, beta labs seems pretty zippy overnight and this morning, very few false failures out of the test builds [18:33:06] will do; and great! [18:33:40] Coren: re [18:35:38] OrenBochman1: you want to poke multichill (Marteen?) when he's around [18:37:26] sure [18:44:54] 3Wikimedia Labs / 3tools: Replicate the Phabricator database to labsdb - 10https://bugzilla.wikimedia.org/50422#c4 (10Andre Klapper) No idea what "Phabricator" database this is about. Everything? Plus no usecase described here at all. Plus I don't see how to sanitize that data (legal). [18:46:54] 3Wikimedia Labs / 3tools: I can't access ptwiki_p database - 10https://bugzilla.wikimedia.org/69701#c5 (10Rodrigo Sampaio Primo) Thanks! [18:48:09] 3Wikimedia Labs / 3tools: Replicate the Phabricator database to labsdb - 10https://bugzilla.wikimedia.org/50422#c5 (10Marc A. Pelletier) I think the original use case (If I recall correctly from IRC) was to get gerrit /and bugzilla/ databases available from labs. I don't think there's a problem in principle... [18:51:39] 3Wikimedia Labs / 3tools: Replicate the Phabricator database to labsdb - 10https://bugzilla.wikimedia.org/50422#c6 (10Kunal Mehta (Legoktm)) (In reply to Andre Klapper from comment #4) > No idea what "Phabricator" database this is about. Everything? > Plus no usecase described here at all. > Plus I don't see... [19:04:57] 3Wikimedia Labs / 3tools: Include pagecounts dumps in datasets - 10https://bugzilla.wikimedia.org/48894#c26 (10Marc A. Pelletier) 5REOP>3RESO/FIX After a couple of odd issues with the underlying filsystem that took us several days to fix, the server is back online with the dumps. The pagecounts aren't /... [19:52:42] The Tools Directory hasn't been updated since the 19th? [20:05:09] 3Wikimedia Labs / 3tools: dewiki / wikidatawiki replication (s5) has stopped on labsdb1001/labsdb1002 - 10https://bugzilla.wikimedia.org/69776#c3 (10merl) The replication had been stopped some minutes after a wikidatawiki db schema change was executed on wmf cluster to enable wikidata as langlinks repository... [21:36:39] 3Wikimedia Labs / 3Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049#c4 (10Antoine "hashar" Musso) The current quotas are: Cores: 25/30 RAM: 51200/102400 Floating IPs: 0/0 Instances: 9/12 Security Groups: 0/10... [21:38:14] 3Wikimedia Labs / 3Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - 10https://bugzilla.wikimedia.org/70049#c5 (10Antoine "hashar" Musso) Or we can just bump core to 40 and instances to 20. Giving us some more room. Andrew: if you need budget to thr...