[00:55:09] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c3 (10Sean Pringle) Please post examples of both queries. [00:55:26] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110 (10Sean Pringle) a:3Sean Pringle [01:05:39] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - 10https://bugzilla.wikimedia.org/69127 (10Sean Pringle) a:3Sean Pringle [01:22:36] 3Wikimedia Labs / 3tools: Packages to be added to toollabs puppet - 10https://bugzilla.wikimedia.org/53704 (10Tim Landscheidt) [01:22:39] 3Wikimedia Labs / 3tools: Is it possible to install w3c-markup-validator? - 10https://bugzilla.wikimedia.org/69105#c2 (10Tim Landscheidt) 5NEW>3ASSI a:5Marc A. Pelletier>3Tim Landscheidt w3c-markup-validator pulls in apache2 which is suboptimal, but the deal breaker would be if that somehow fires up... [02:29:09] 3Wikimedia Labs / 3wikitech-interface: Remove Puppet class generic::packages::git-core and replace misc::package-builder with role::package::builder::labs - 10https://bugzilla.wikimedia.org/69135 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None The class generic::packages::git-core doesn't exist... [02:29:52] 3Wikimedia Labs / 3wikitech-interface: Remove Puppet class generic::packages::git-core and replace misc::package-builder with role::package::builder::labs - 10https://bugzilla.wikimedia.org/69135#c1 (10Tim Landscheidt) And in both cases it would probably be nice to see if there are instances with those class... [02:38:14] Is there a known issue with Beta Labs updating to master? [02:38:40] GettingStarted is a couple of days old on Beta Labs enwiki [02:38:45] http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [03:48:28] superm401: check that the mediawiki-extensions meta repo has an up to date version of GettingStarted? [03:48:52] legoktm, my understanding is Beta Labs is supposed to update to master. [03:49:24] yeah, and it uses the mediawiki/extensions meta repo to do so [03:49:32] https://github.com/wikimedia/mediawiki-extensions [03:49:44] > GettingStarted @ a5fe23d [03:50:14] looks up to date [03:51:22] legoktm, hmm, thanks, yeah, that's the latest, but Beta Labs is not at a5f. [07:29:40] Hello. Does anyone know how can I filter out protected pages from a mediawiki API query? [10:03:07] 3Wikimedia Labs / 3Infrastructure: eqiad instances are not showing in labs ganglia - 10https://bugzilla.wikimedia.org/62693#c1 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX That was some cruft related to the pmtpa -> eqiad migration. Ganglia on labs is dead but that is a different issue. [11:40:52] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c4 (10Incola) The query is at the bottom of the previous link. The query that fails is not important, the problem is that this query takes too long to run. [11:47:25] Coren_away / andrewbogott_afk / springle: mariadb10 server using s5.labsdb/s2.labsdb currently unreachable [11:49:24] ganglia reports nearly no server load, no io (so replication must have stopped, too) [11:49:35] Merlissimo, is it the same trouble as "Can't connect to MySQL server on 'commonswiki.labsdb" ? [11:49:48] phe: yes [11:51:36] problem is that all labs admins i pinged are asleep because of Minneapolis/Minneapolis/San Fransicotimezone [12:13:53] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10merl) [12:13:54] 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144 (10merl) 3NEW p:3Unprio s:3blocke a:3None Since 11:42 UTC the database server s2/s4/s5 is unreachable and ganglia shows no io for this server. [12:14:21] 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144 (10merl) p:5Unprio>3Immedi [12:18:17] Merlissimo: here now. checking it [12:26:57] springle: if it help mointor link is https://ganglia.wikimedia.org/latest/?r=2hr&cs=&ce=&c=MySQL+eqiad&h=labsdb1002.eqiad.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS [12:28:45] was a segfault. should be back up, but still investigating stack trace [12:28:51] * Merlissimo already got 259 sge reschedule notification mails [12:31:22] 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144#c1 (10Sean Pringle) segfault. System is back online. Still investigating. [12:31:38] 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144 (10Sean Pringle) a:3Sean Pringle [13:06:07] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c5 (10Sean Pringle) I realize you think the speed is the problem, which I agree is an issue. However there is no "over 600s" type kill mechanism, so I'm interested in establishing two... [13:10:37] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c6 (10Andre Klapper) Incola: "not important" doesn't really exist when trying to find steps to reproduce. :) [14:18:22] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c7 (10Incola) The second query is something like: insert into `executions` (`query_id`, `time`, `duration`, `results`) values (`23`, `2014-08-05 14:01:05`, `1879`, `5290`) [14:25:07] 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144#c2 (10Sean Pringle) The crash was similar to https://mariadb.atlassian.net/browse/MDEV-6455, with same assertion failure but a DELETE stack trace instead of UPDATE. Needs more research including labsdb1... [16:01:19] Hey folks. I might just be jet lagged, but I can't seem to see any of my instances under "Manage instances". [16:01:28] Something change in the last month or so? [16:10:26] halfak: hey! [16:10:28] halfak: that's a recurring bug. log out and back in [16:10:30] halfak: ALSO HI [16:11:20] Hi. Are you @ the hotel? [16:11:45] halfak: no, with my friend, about 1h away. I'll be at the hotel in a couple of hours [16:12:03] Hokay. :) How's the jet lag? [16:12:26] halfak: not much at all. I was going to sleep at 5am - 7 AM in India, so going to sleep at midnight here feels natural ;) [16:12:45] halfak: my body gave up on using light/dark cycles to synchronize sleep a looooong time ago [16:12:59] I'm jealous. [16:13:04] halfak: as the Hulk says in The Avengers, "I'm always jetlagged" :) [16:13:33] halfak: heh :) [16:18:01] * halfak is building his instance :) [16:40:20] I just built a large instance. How do I get my hands on the 160GBs that comes withit? [16:42:21] halfak: It's available via LVM. Either you can do it yourself or use role::labs::lvm::mnt. [16:43:00] thanks scfc_de ! [16:44:03] scfc_de, do you know if there is a role I can enable to pull XML database dumps from a shared mount? [16:44:20] I see that there is a "/public/dumps" folder that is unmounted. [16:45:32] halfak: That should be the default IIRC as role::labs::instance mounts that. But I think that dumps is broken at the moment due to the move to a new server or something. [16:46:07] Gotcha. I guess I'll just have to pull 'em down [16:46:16] halfak: What does "mount | fgrep dumps" say on your instance? [16:46:38] Nothing [16:46:48] Hmmm. And Puppet runs clean? [16:47:23] Not sure. How do I ask it to run? [16:47:56] sudo puppet agent apply -tv [16:48:46] "notice: Run of Puppet configuration client already in progress; skipping" [16:48:50] Actually, the "apply" is superfluous there => "sudo puppet agent -tv". [16:49:12] halfak: Then the initial Puppet run is still happening and as part of this, the dumps mount will be mounted later on. So: Patience. [16:49:27] Hokay. Thank you. [16:49:42] halfak: You could look at /var/log/puppet.log for the current running Puppet. [16:51:20] halfak: And https://bugzilla.wikimedia.org/show_bug.cgi?id=66362 is the bug about dumps not working 100 % at the moment. [16:53:14] Yikes. That bug... [17:43:55] I know everyone is busy with Wikimania but could a Labs admin approve my project? Thanks. [17:44:07] https://wikitech.wikimedia.org/wiki/New_Project_Request/Performance [19:19:05] beta labs is returning immediate HTTP 503s for api.php, load.php, and wiki pages [19:19:24] e.g. http://en.wikipedia.beta.wmflabs.org/w/index.php [19:20:20] is anyone around to restart them? Coren_away , bd808|BUFFER , hmmm :) [19:21:55] spagewmf: ack, fixing [19:26:54] spagewmf: back up [19:27:38] ori: Looks good [19:28:13] ori: Looks like http://deployment.wikimedia.beta.wmflabs.org/ is in a redirect loop [19:50:22] Is there a way to check what software is installed on the Tool Labs job execution instances (i.e. the instances where the jsub jobs are run)? [19:50:33] I'm trying to figure out if npm is installed. [19:55:15] Separately, Beta Labs is still not updating to latest master for GettingStarted. [19:55:33] Is anyone else having that problem? [19:55:43] npm is installed [19:55:48] but it's old [19:56:03] you can check by sshing to one of the exec nodes [19:56:15] or just try running a job and see what happens [19:56:29] jeremyb, what's the name of a node I can ssh to? [19:58:35] superm401, e.g. ssh tools-exec-03.eqiad.wmflabs [20:01:15] jeremyb, thanks. However, the npm Debian package does not seem to be installed. If it is installed some other way, do you know where it is? [20:01:19] which npm and which npmjs give nothing. [20:01:49] I see nodejs is installed, but not npm. [20:03:00] huh [20:03:07] well npm is on tools-login [20:03:14] why do you need it on an exec node? [20:03:51] (i'm running something now on grid that was installed with npm on tools-login) [20:05:43] jeremyb, I'm debugging a script that uses it. It probably doesn't actually need it. [20:06:27] ok :) [20:44:04] Has anyone installed MW in Tools? [20:57:39] 3Wikimedia Labs / 3tools: MediaWiki blank page after uploading LocalSettings.php - 10https://bugzilla.wikimedia.org/69154 (10Jamison Lofthouse) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier In Tools I have a MediaWiki installation setup in my /public_html/mediawiki directory. Setup of the installati... [21:36:24] I added python code to compat, I can see it in WinSCP, but when I give a command at putty this message appears "[Errno 2] No such file or directory". how to fix it? [21:37:03] Hi garbak6 [21:37:11] garbak6: what command specifically are you entering in PuTTY? [21:37:35] (that is, I presume that you are ssh'ing in via PuTTY to the command line within your labs instance or tool on labs, right?) [21:38:21] this is the command: python boxfinder.py -cat:Academy_Awards_ceremonies -enonly. [21:39:55] or: python boxfinder.py -cat:Bovines -enonly. not the first one, sorry [21:40:31] it gets data from infoboxes in enwiki [21:41:48] garbak6: so, did this problem just start when you changed boxfinder.py ? [22:16:22] 3Wikimedia Labs / 3tools: MediaWiki blank page after uploading LocalSettings.php - 10https://bugzilla.wikimedia.org/69154#c1 (10Andre Klapper) Probably username info might be welcome to investigate further; also has this been brought up on the mailing list or #wikimedia-labs IRC before to do some more debugg... [22:18:07] 3Wikimedia Labs / 3tools: Failed to set group members for local-oclc-reference - 10https://bugzilla.wikimedia.org/65534#c4 (10Andre Klapper) 5UNCO>3RESO/WOR (In reply to Andrew Bogott from comment #3) > Please verify that this is working for you as well, and if it looks ok then > mark this bug as 'worksf... [22:30:22] 3Wikimedia Labs / 3tools: User accounts should have a replica.my.cnf - 10https://bugzilla.wikimedia.org/57485#c4 (10Andre Klapper) Yuvi: ping - please reply before closing your ticket as not actionable. Thanks. [23:44:37] 3Wikimedia Labs / 3tools: MediaWiki blank page after uploading LocalSettings.php - 10https://bugzilla.wikimedia.org/69154#c2 (10Jamison Lofthouse) This has been brought up on #wikimedia-labs yesterday and Sunday and no one could figure out what was wrong. We explored permissions, Lighttpd, PHP, Apache, and t...