[01:40:22] Hey scfc_de. I was bugging you about 6 hours ago with an instance where puppet didn't finish. Looks like it still hasn't. Any suggestions? [01:40:32] *7 hours ago [01:41:42] halfak: Sure Puppet is *still* running and not one of the half-hourly runs since? [01:42:05] I guess that is a fine question. however I still don't have my mount point for storage. [01:42:36] If you run "puppet agent -tv", what does it say? [01:43:32] Instance list reports puppet status as "failed". [01:44:29] notice: Run of Puppet configuration client already in progress; skipping [01:45:55] What does the "end" of /var/log/puppet.log say? [01:58:08] http://pastebin.com/gVup7xng [01:58:28] YuviPanda, just told me that I really want lvm::srv not lvm::mnt [02:03:22] halfak: So the mount is now there? [02:03:30] No. [02:04:36] YuviPanda, "it is stuck at trying to mount dumps. I think the new dumps nfs server is kinda active, the old one is full and stuck, and puppet nfs is confused as fuck and doesn't complete. hangs forever" [02:04:53] tryinng to create lvm by hand now [02:10:04] halfak: But AFAIK "notice: /Stage[main]/Role::Labs::Instance/Mount[/public/dumps]/ensure: ensure changed 'unmounted' to 'mounted'" means that the mount actually succeeded. [02:10:23] Whatever was happening after that failed. [02:10:41] Also, neither /public/dumps nor /srv seems to be mounted and available. [02:14:39] Hmmm. Odd. [02:20:24] We have manually mounted the directory by running a few commands from puppet. [02:21:10] This'll work with me for now. How do I report my issues in a useful way? [02:25:16] Wikimedia Labs => General. Re "useful" however, pretty hard to make something reproduce because many cogs are involved :-). [02:26:16] Well, it is a fresh instance. [02:27:03] Step (1): Start new instance "large". Step (2): Check ...::lvm::srv. Step (3): Wait and be sad. [02:27:51] :-) [02:30:23] https://bugzilla.wikimedia.org/show_bug.cgi?id=69161 [02:30:24] 3Wikimedia Labs: /srv won't mount on new 'large' instance - 10https://bugzilla.wikimedia.org/69161 (10Aaron Halfaker) 3NEW p:3Unprio s:3normal a:3None I started a new instance ('large') and selected role::labs::lvm::srv, however the puppet script hangs (for hours) after trying to mount the storage. Re... [02:30:29] Thanks for your help scfc_de [02:30:34] ha! In before wikibugs [02:30:35] np [08:24:11] YuviPanda: you here? that new stream doesn't seem to work I get websocket._exceptions.WebSocketException: Handshake status 502 [08:24:22] for https://wikitech.wikimedia.org/wiki/RCStream [08:24:32] hashar ^ [08:24:42] this really needs to work before you phase out IRC feed [08:27:11] petan: is that on beta or prod? [08:27:17] prod [08:27:25] I just use that python sample [08:28:01] I don't even know whether the prod one is enabled :D [08:28:37] ... [08:28:41] petan: poke _joe_ about it in #wikimedia-operations [08:29:06] I think he is the one from ops that helped ori set it up [08:34:59] hashar: now I get http://deployment.wikimedia.beta.wmflabs.org/wiki/Meta:Huggle/List The page isn't redirecting properly [08:35:08] Firefox has detected that the server is redirecting the request for this address in a way that will never complete. This problem can sometimes be caused by disabling or refusing to accept cookies. [08:36:37] there is a redirect loop :/ [08:37:06] $ curl --verbose -L http://deployment.wikimedia.beta.wmflabs.org/wiki/Meta:Huggle/List 2>&1 |fgrep Location [08:37:06] < Location: http://deployment.wikimedia.beta.wmflabs.org/wiki/Meta:Huggle/List [08:37:06] < Location: http://deployment.wikimedia.beta.wmflabs.org/wiki/Meta:Huggle/List [08:37:06] < Location: http://deployment.wikimedia.beta.wmflabs.org/wiki/Meta:Huggle/List [08:37:06] .... [08:39:06] what is: petrb@huggle:/tmp$ python test.py [08:39:08] No handlers could be found for logger "socketIO_client" [08:39:11] o.O [08:39:29] ok so I guess that this stream thing not returning anything is normal when beta is broken :P [08:43:04] !log deployment-prep prunning cache on deployment-cache-text02 / restarting varnish [08:43:07] Logged the message, Master [08:43:41] pfff [08:46:49] must be something on varnish side :-/ [08:50:30] or on apache [08:50:33] no clue really :-( [09:50:19] !log integration upgraded package on trusty node hhvm-dev bumped (3.3-dev+20140728+wmf3) over (3.3-dev+20140728+wmf1) [09:50:21] Logged the message, Master [10:36:29] Coren_away: andrewbogott_afk increase quota for project quarry? [10:39:24] hasharEat: petan RCStream isn't prod yet [10:39:33] hasharEat: petan it's being worked on, not announced as 'prod' yet [10:47:33] hmmmm [10:47:43] root@ogvjs-testing:~# /sbin/start jobrunner [10:47:44] start: Job failed to start [10:48:14] lemme update mediawiki [10:48:27] still fails [10:53:17] for you YuviPanda :D https://bugzilla.wikimedia.org/69169 [10:55:52] petan: rcstream.wmflabs.org is relatively stable [11:00:23] aha my /vagrant was out of date [11:00:27] lemme see if that helps [11:02:39] yep that helped \o/ [11:02:42] ok closed out bug [11:02:59] halfak: I'm at the nearby table, helping someone set up on ToolLabs [11:05:44] YuviPanda: *waves* [11:05:50] (not physically waving) [11:06:22] * YuviPanda waves at valhallasw`mania [11:06:26] valhallasw`mania: oh, where are you at? [11:06:33] YuviPanda: upstairs [11:06:43] next to the coffeeeeeeeee [11:06:48] it’s still humid downstairs [11:06:52] and lunch isn’t til 1 [11:06:52] valhallasw`mania: ah, I'm inside, helping someone set up on labs [11:07:14] i got my labs-vagrant updated and fixed it all by myself! i’m proud [11:07:36] I'm downstairs, also next to the coffeeeee (and teaaaa), but close to the door leading to the smoking area. :-) [11:08:52] Coren_WM2014: ah, cool! Is there a toollabs table anywhere? if not there's one by the lunch now [11:08:59] Coren_WM2014: also I need more quotaaaaaaa! [11:09:41] YuviPanda: I should probably join you though, there's no designated area here. (But smooooking!). Join me for a smoke downstairs and I'll head back with you up there. :-) [11:09:56] Coren_WM2014: ah :) am helping someone right now, I'll be there in a bit [11:10:20] This building is too confusing :-p [11:22:36] YuviPanda: well, I'm down there now, too [11:37:26] YuviPanda: Have you forgotten me? [11:39:15] YuviPanda: WHY HAVE YOU FORSAKEN US?????//////slashslashslash [11:59:07] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/talkcatintersect.php to Tool Labs - 10https://bugzilla.wikimedia.org/60874#c2 (10Andre Koopal) 5NEW>3PATC I did some fixes that it looks at the correct databases for the category cache. If somebody can do some more checks if it wo... [12:03:53] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c11 (10Andre Koopal) 5ASSI>3RESO/FIX I tried today, and the simple version is working now as well. Not sure if something changed, but closing off this one. [12:06:07] 3Wikimedia Labs: /srv won't mount on new 'large' instance - 10https://bugzilla.wikimedia.org/69161#c1 (10Tim Landscheidt) Note that the/an issue with /public/dumps may have been fixed by Gerrit change #152022. [12:07:30] !log deployment-prep Apache virtual hosts were not properly loaded on mediawiki02. I have hacked /etc/apache2/apache2.conf to make it Include Include /usr/local/apache/conf/all.conf (instead of main.conf which does not include everything) [12:07:32] Logged the message, Master [12:08:36] !log deployment-prep Manually pruning whole text cache on deployment-cache-text02 [12:08:39] Logged the message, Master [12:39:49] Coren_WM2014: Can we create a new labs project? [12:42:30] Reedy: Whereabouts are you? [12:42:34] YuviPanda: Also you? [12:46:41] Coren_WM2014: I'm near the projector [12:46:45] Coren_WM2014: with the researchers! [12:46:48] Coren_WM2014: come? [12:47:07] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c8 (10Sean Pringle) The first query is heavily dependent on disk IO. It runs in ~1000s on both MariaDB 10 and 5.5 if data is cold, or if any other concurrent query is also bottle necke... [12:53:24] Pywikibot course in Frobisher 4-5 in 10 minutes! [13:13:24] !log deployment-prep removed a bunch of local hack on deployment-bastion:/srv/scap-stage-dir/php-master . That causes the git repo to be dirty and prevents scap from achieving git pull there [13:13:26] Logged the message, Master [13:13:58] !log deployment-prep https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is running again [13:13:59] Logged the message, Master [13:51:39] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - 10https://bugzilla.wikimedia.org/69182 (10metatron) 3UNCO p:3Unprio s:3critic a:3None When accessing INFORMATION_SCHEMA, query hangs up on random schemas and causes a widespr... [14:01:22] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - 10https://bugzilla.wikimedia.org/68753#c7 (10metatron) See also https://bugzilla.wikimedia.org/show_bug.cgi?id=69182 [15:34:22] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - 10https://bugzilla.wikimedia.org/69110#c9 (10Incola) - The first query always runs correctly. - They are on different connections. - I don't know because I'm not the original author of the code and I don't know how works... [15:34:36] 3Tool Labs tools / 3Erwin's tools: Erwin's related changes doesn't handle wikidata edits properly - 10https://bugzilla.wikimedia.org/66874#c3 (10Andre Koopal) 5NEW>3RESO/FIX Hackatons are great, was able to make some time, and got the needed info on how wikidata edits are stored in the recentchanges tabl... [17:05:25] 3Wikimedia Labs / 3tools: MediaWiki blank page after uploading LocalSettings.php - 10https://bugzilla.wikimedia.org/69154 (10Jamison Lofthouse) [17:09:44] I updated my server to the latest labs-vagrant version... now I can no longer view any pages [17:10:03] I get a ERROR 500: hphp_invoke error [17:14:33] physikerwelt: Did you have any errors from `labs-vagrant provision`? [17:15:38] no not at all [17:16:14] bd808: I'll rerun puppet [17:21:22] bd808: OK The problem is gone again. After several puppet runs it showed that a new role had been enabled labs_initial_content ... I don't remember to enable this role [17:22:04] physikerwelt: That role gets enabled by the labs::vagrant role on the host. [17:23:17] bd808: OK I checked again... the error was only temporary gone [17:25:39] physikerwelt: glad it cleared up. Shout if it comes back [17:27:17] Is anyone an admin on labs? [17:28:19] bd808: could you try wget http://drmf.beta.wmflabs.org/wiki/Main_Page [17:28:45] physikerwelt: I get "ERR_CONTENT_DECODING_FAILED" [17:29:37] bd808: I think there is a php-error somewhere... but the error message does not help to figure out what is wrong [17:29:57] Yeah. hhvm is puking on something [17:30:07] check /var/log/hhvm/error.log [17:31:27] bd808: Thanks a lot... that's exactly what I was searching for since hhvm has been introduced [17:31:47] excellent [17:39:39] Is there a Labs admin? [17:44:46] (03PS1) 10Multichill: Some Wikidata crap I wanted to share [labs/tools/multichill] - 10https://gerrit.wikimedia.org/r/152147 [17:45:13] Negative24: what for [17:45:28] Approving a project request. [17:46:06] (03CR) 10Multichill: [C: 032 V: 032] "Want to share it right away" [labs/tools/multichill] - 10https://gerrit.wikimedia.org/r/152147 (owner: 10Multichill) [17:47:24] petan: https://wikitech.wikimedia.org/wiki/New_Project_Request/Performance [17:57:51] motd [17:58:47] Negative24: andrewbogott_afk and Coren_WM2014. [17:59:32] scfc_de: I guess that's why my request has been waiting. Everyone is at Wikimania.