[03:27:12] The team sizes of Triple-A gaming these days makes Super Mario World look indie [03:27:55] <^demon|away> I should replay Super Mario World. [03:28:46] So should I. [03:28:53] In fact, I should buy a SNES so I can play it legit. [03:28:54] You don't have a steam pile of shame? [03:30:30] <^demon|away> Heh, I definitely have a pile of shame. [03:30:32] Shame? Nope, none of that. [03:30:47] <^demon|away> I never get through it because I always get distracted by the SNES and so forth :p [03:31:22] I finally gave in and bought CoD4 for $10 to finally play Activision's darling child [03:32:00] Would you like to start? [03:32:41] https://www.humblebundle.com/sgdq [03:35:51] http://nintendo.wikia.com/wiki/Super_Mario_World/credits 17 people worked directly on SMW [03:35:59] <^demon|away> Heh, nobody's trying to marathon Earthbound. [03:37:04] <^demon|away> Which I also need to replay. [03:38:27] No Man's Sky is over hyped (Re-imagined Spore), likely one of the 10 developers worked in marketing [03:39:29] <^demon|away> Heh, the NMS logo reminds me of Din's Fire in Ocarina of Time. [03:46:48] i actually can't wait for no man's sky. [04:01:29] Apparently I grew up with a different definition of science fiction. Mine was What-If some fundamental was different, how would the world be different. The correct definition: Let explore contemporary drama and politics in space. The What-If is the procedural generated game I'd like to explore. [04:06:14] Something set in an era I never really lived in. Like Shenmue's 1980s [04:10:11] * Dispenser waves the team good night [06:09:47] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c8 (10Sean Pringle) Federated filearchive, filearchive_userindex, user_daily_contribs, user_properties should now exist, pointing to the sanitized labsdb1002 commonswiki_p versions. Need to g... [08:18:03] !ping [08:18:03] !pong [08:18:05] gah [09:36:34] hashar: around? I've setup charcoal.wmflabs.org to be diamond + graphite. I need to test https://gerrit.wikimedia.org/r/#/c/142210/ on something with self hosted puppetmaster before it can get merged, and am wondering if I can test that on the betalabs puppetmaster. [09:36:37] can you give me access? [09:37:34] don't you have root access on beta already? :D [09:37:52] hashar: no, it got removed at some point. [09:38:48] !log deployment-prep Granting sudo to YuviPanda [09:38:50] Logged the message, Master [09:38:56] YuviPanda: the puppet master is deployment-salt.eqiad.wmflabs [09:39:01] hashar: woot! going now. [09:39:01] under /var/lib/git/operations/puppet [09:39:06] beware there is a bunch of cherry pick there [09:39:13] that you don't want to erase. So please please be careful [09:39:26] hashar: yeah, I'll do a cherry-pick as well [09:39:38] as I understand it, the diamond on labs send its metric to prod and that is rejected/firewalled. So it is fine sending the metrics somewhere else [09:40:02] if you have anything worthwhile to announce, you might want to announce it on labs-l [09:40:09] hashar: yeah, I will once this gets merged :) [09:40:11] ie 'experimenting diamond on labs' [09:40:24] I am sure bd808 will be quite happy about it as well as chase / andrewboggot and coren [09:42:33] hashar: will do! :) you can check out charcoal.wmflabs.org (username/pw: guest/guest) [09:42:39] I want to eventually make it fully public [09:42:47] and point only public events there [09:42:54] nginx stats from the proxy being the first thing, for example. [09:43:50] \O/ [09:44:44] hashar: cherry-picked :) Made sure to not lose what was in the puppetmaster already [09:44:52] good job! [09:44:58] and !log :-] [09:45:13] hashar: yes [09:45:22] have you considered namespacings the metrics by project name? [09:45:33] with all the instances we have on labs, the tree might ends up hard to browse [09:45:40] !log graphite created diamond-collector with roles diamond and graphite and pointed charcoal.wmflabs.org to it [09:45:42] Logged the message, Master [09:46:00] !log deployment-prep cherry-picked https://gerrit.wikimedia.org/r/#/c/142210/ on to puppetmaster [09:46:02] Logged the message, Master [09:46:34] running puppet on bastion and apache{01,02} [09:46:41] bah apache is broken :] [09:46:57] -host = 10.64.0.18 [09:46:57] +host = 10.68.17.169 [09:46:58] \O/ [09:47:15] does a change to /etc/diamond/handlers/StatsHandler.conf trigger a refresh of diamond? [09:47:38] apparently not [09:47:55] !log deployment-bastion restarted diamond [09:47:56] deployment-bastion is not a valid project. [10:00:03] 3Wikimedia Labs: centralauth_p is missing tables - 10https://bugzilla.wikimedia.org/66533#c1 (10Silke Meyer (WMDE)) a:3Marc A. Pelletier Hi Coren, 5 days to go... Please say something on this! [10:01:04] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c9 (10Silke Meyer (WMDE)) a:5Sean Pringle>3Marc A. Pelletier Thanks Sean and ping Coren! [10:06:54] hashar: hmm, interesting. I see data coming in, but nothing on graphite [10:08:22] diamond also probably needs more aggressive logrotate [10:09:01] YuviPanda: no clue how that works. You are on your own ! [10:09:02] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c4 (10Silke Meyer (WMDE)) When I try it now, it also seems to work for commons. How exactly can I see what isn't working? [10:09:08] hashar: yeah :) [10:09:14] hashar: just talkign alound, I'll do that without pinging you now [10:09:30] yeah just send a summary to labs-l once you have something [10:09:41] if blocked, you might want to reach out to ops (chase iirc) [10:10:20] yeah, will do :) [10:17:17] chasemp: got a minute? [11:10:03] 3Wikimedia Labs: centralauth_p is missing tables - 10https://bugzilla.wikimedia.org/66533#c2 (10Andre Klapper) CC'ing springle as this is getting urgent (abogott is already CC'ed). [12:01:50] !ping [12:01:50] !pong [12:48:51] !log deployment-prep cherry picked https://gerrit.wikimedia.org/r/#/c/142228/ to puppetmaster, sending events to charcoal.wmflabs.org now with projectname \o/ [12:48:54] Logged the message, Master [13:05:14] Hello. I'm getting "ERROR 1430 (HY000): There was a problem processing the query on the foreign data source. Data source error: error: 1054 'Unknown column 'page_content_model' in 'field list'" at queries on labs DBs. Is there any update missing? [13:06:09] YuviPanda: charcoal looks nice [13:06:26] YuviPanda: next step: generates gdash dashboards out of the metrics hehe [13:07:51] hashar: yeah, but I'm also looking at non-gdash alternatives. [13:07:53] hashar: look at http://grafana.org/ [13:08:34] Besides Graphite or InfluxDB, it only has one optional external dependency and that is Elasticsearch. [13:08:35] ohh [13:08:43] though optional [13:08:44] we have evaluated grafana [13:08:51] and we'll install it probably [13:08:54] woot! [13:08:59] there's even https://gerrit.wikimedia.org/r/#/c/133274/ [13:09:58] paravoid: woot. that's nice. [13:10:18] YuviPanda: you should talk to bd808 . He show me a python based frontend to graphite that was quite nice (and python) [13:11:03] paravoid: I just setup diamond+graphite for labs, and seems to work fine (waiting for merges). next is to add more important metrics for tools, such as nginx stats, grid engine stats, redis stats, etc. [13:11:09] paravoid: thanks for pointing out grafana patch! [13:11:33] \O/ [13:12:01] hashar: will talk to bd808|BUFFER when he's around :) hopefully I can get the two patches merged today and then email labs-l. [13:12:20] hashar: I also want to remove the auth before charcoal. shouldn't have any private info [13:14:19] until people start using diamond to do so :D [13:14:43] hashar: true, but if we fully publicize the fact this is public, then... :) [13:15:27] YuviPanda: yeah I am teasing you sorry [13:15:33] hashar: :D i know :) [13:22:53] Coren: around? [14:07:29] @notify andrewbogott [14:07:29] I'll let you know when I see andrewbogott around here [14:12:31] anyone in here familiar with openstack enough to know if the whole world breaks if you do "restart nova-network"? :) [14:12:45] that command seems to be how one is supposed to get new dnsmasq config loaded, but it sounds a little scary [14:12:58] bblack: what's the worst that can happen? :) [14:13:18] It could be full of racist ninjas [14:13:53] bblack: better than racist samurais? [14:16:05] :) [14:16:19] that was from: http://www.youtube.com/watch?v=20KJhBX9xtE [14:16:27] (took me a while to find a link to that scene) [14:22:19] YuviPanda: I'm up and about now :) commented on https://gerrit.wikimedia.org/r/#/c/142228 [14:32:20] * YuviPanda pounces on andrewbogott :) [14:32:25] hi! [14:32:39] YuviPanda: Am now. [14:32:58] (Sorry, still on Alaska time) [14:33:01] Coren: andrewbogott charcoal.wmflabs.org (guest/guest). graphite for labs via diamond. [14:33:49] Coren: andrewbogott patches haven't been merged yet, but receiving data from betalabs. https://gerrit.wikimedia.org/r/#/c/142228/ and https://gerrit.wikimedia.org/r/#/c/142210/2 [14:34:01] bblack: restarting nova-network is OK -- at worst it will cause a brief labswide newtwork hiccup, but shouldn't do that even. [14:34:20] YuviPanda: um… give me a minute [14:34:52] andrewbogott: ok, giving it a spin! [14:35:43] !log restarted nova-network on labnet1001 [14:35:43] restarted is not a valid project. [14:35:49] lol [14:42:14] * YuviPanda waves at bd808 [14:44:12] Hey YuviPanda. I was just reading about charcoal in backscroll [14:44:31] bd808: :) I should probably rename it, considering there's also charcoal a graphite frontend... [14:47:37] YuviPanda: I started to setup graphite in beta quite a while ago but got distracted by other work and never fully built out the stack. There is a graphite server at https://graphite-beta.wmflabs.org/ with txstatsd running as well but no gdash. [14:47:55] bd808: right. was thinking of just keeping graphite itself open, perhaps with grafana.org [14:48:02] bd808: since I don't think there'd be any sensitive data in this one [14:48:09] * bd808 nods [14:48:36] I think it's only sql stuff in the prod one that makes it sensitive [14:48:42] bd808: yeah. [14:48:51] bd808: I should also get rid of the auth stuff for labs in the role [14:55:33] Coren, Silke_WMDE, are there any toolserver-migration issues that I'm blocking or that I can work on? [14:56:21] andrewbogott: I'm still catching up atm; I see one DB issue that is blocking but that'll need Sean. [14:56:39] Coren, yep, he just emailed about that I think [14:56:54] Hi andrewbogott and Coren! I have some db issues on my mind as well... one where Sean said you are needed, Coren. [14:57:09] that's this one: https://bugzilla.wikimedia.org/show_bug.cgi?id=59683 [14:57:11] Silke_WMDE: Yeah, they need us both. :-) [14:57:27] then here: https://bugzilla.wikimedia.org/show_bug.cgi?id=66533 [14:57:29] Silke_WMDE: Those are at the top of my TODO for today. [14:57:37] ok [14:58:10] Hope you had a good vacation, Coren!? [14:58:28] Silke_WMDE: I had. The return to Real Life is rough though. :-P [14:58:29] andrewbogott: btw, I found a way to replay our logs on to an nginx server to simulate load (a httperf driver). Should find time to do it sometime next week. [14:58:38] Coren: yeah, I can imagine [14:59:05] YuviPanda: that's great! An issue came up with that local repo last week, so now I'm extra motivated to get rid of it. [14:59:15] although maybe there's other stuff in there that we need besides nginx... [14:59:42] andrewbogott: yeah, lots :D we can get rid of it in dynamicproxy tho [14:59:47] Coren: andrewbogott Will any of you be available online this weekend, in case people start last minute migrations? [15:00:10] Silke_WMDE: I'll be around. [15:00:18] I'll be partly online but I can't really help them. [15:00:26] hashar: so, the DNS thing seems to work, at least for the case I cared about with mobile zero-portal fetches [15:00:44] Silke_WMDE: I can be a bit on Sunday, not so much on Saturday. [15:01:08] bblack: deployment-apache01:~$ dig +short en.wikipedia.beta.wmflabs.org [15:01:08] 10.68.16.16 [15:01:30] is that a good result or a bad one? :) [15:01:30] bblack: seems to fulfill the need. Might want to announce the nice trick on labs-l mailing list since I am sure other will be interested [15:01:36] bblack: a good one [15:01:45] bblack: it uses to reply back with the public IP [15:01:50] right, ok [15:02:09] so, wait a few days, then kill off natfix? [15:02:23] bblack: I guess so. [15:02:28] I guess could audit iptables rule hit counts on those and see if they're still used after the DNS update, too [15:02:42] YuviPanda: wow, graphite sure does have a lot of nested folders. Is there some kind of overview page like we have in ganglia? [15:02:54] andrewbogott: aah, so... :) [15:02:57] bblack: ah that is beyond my capabilities :-/ [15:03:05] andrewbogott: that's what we're going to need next. those need to be defined. [15:03:12] ok [15:03:16] andrewbogott: looking at grafana.org for that [15:03:18] both in labs and prod [15:03:27] andrewbogott: in the meantime, you can make custom graphs and save them as user graphs [15:03:35] hashar: any pointers on how, as root, to do a dsh/salt command across all the betalabs stuff and from where? [15:03:45] I'm not very familiar with graphite, is it ~ similar to ganglia, or solving a different problem? [15:03:57] or I guess even as not-root would work [15:03:59] bblack: our salt master is deployment-salt.eqiad.wmflabs [15:04:12] ok [15:04:14] andrewbogott: a superset of ganglia, I think. [15:04:26] 'k [15:04:32] andrewbogott: there's a WIP patch https://gerrit.wikimedia.org/r/#/c/133274/ [15:07:08] bblack: I have pinged Wikidata folks that needed the same trick. That will be some more testers added to the loop [15:08:55] hashar: ok [15:09:18] re salt + iptables, basically this tells you rule usage counts: salt '*' cmd.run 'iptables -vnL -t nat|grep DNAT|grep -v "0 0"' [15:09:43] maybe I can reset the stats so that further usage is more obvious, instead of trying to compare current + future numbers [15:12:43] yeah I zeroed out the nat table counters, so now the above salt command won't output anything from any of the hosts if things are working as expected [15:12:55] if someone the nat rules get used, hits should show up [15:13:26] (e.g. maybe someone's got something that's using the public IP address directly without DNS, and then routing through natfix) [15:18:21] hashar: does this affect tool labs as well, or just deployment/beta stuff? [15:19:05] bblack: yeah just zeroing the iptables counter is fine. Nobody looks at them I guess :] [15:19:18] bblack: the issue affect anyone on labs because of NAT [15:19:28] so your neat dns alias also the fix the issue for everyone [15:19:53] a nice side effect is that labs instance can now use any DNS entry that points to the shared reverse webproxy [15:20:25] that is a huge improvement to end users [15:21:56] you should probably write any email to labs-l, I have no idea about any of this except "hey this fixed my personal problem" :) [15:23:18] hashar: going to remove my cherry-picks from betalabs since they got merged. should I just remove them and rebase? [15:27:16] YuviPanda: I think bd808 has a cron that automatically rebase from time to time [15:27:19] that might pick it up [15:27:31] hashar: ah, ok. other issues, actually (dependent patch forgotten to be +2'd, fixing now) [15:29:02] YuviPanda, hashar: The cron on deployment-salt rebases once an hour. It should clean up most merged cherry-picks but occasionally needs manual intervention. [15:30:07] \O/ [15:30:32] bd808: meanwhile, I have my zuul cloner that tests core + a bunch of extensions. I find a bunch of bugs running it :-] [15:30:59] bd808: for vendor, we will need the libraries to be merged first before being able to test patches proposed to core. I think I dropped you a rather long email about it [15:31:18] Nice. ori has been trying to run *all* unit tests at once for hhvm testing. He said he filed a bunch of bugs too. [15:32:34] hashar: I'll see if I can find someone WP:BOLD enough to review and merge my vendor patches soon [15:33:39] https://integration.wikimedia.org/ci/job/mediawiki-core-extensions-integration/37/console [15:33:41] example run [15:33:53] I have stolen your color codes from scap/scap/scap/log.py :] [15:34:16] that job takes core + almost all extensions from current wmf branch [15:34:18] then run phpunit [15:34:19] and dies [15:34:34] the clones are made with zuul cloner, so it can get a specific patch from gerrit [15:34:40] I guess we could run hhvm as well [15:35:36] !ping [15:35:37] !pong [15:38:25] hashar: That's a neat jenkins job. ori, check out https://integration.wikimedia.org/ci/job/mediawiki-core-extensions-integration/37/consoleFull when you've got time :) [15:42:13] that is evil plan 2 [15:43:32] bd808: for core / vendor the patch is https://gerrit.wikimedia.org/r/#/c/141819/ that generates https://integration.wikimedia.org/ci/job/mediawiki-core-vendor-integration/ [15:44:04] can be triggered on a change to mediawiki/core or mediawiki/core/vendor by commenting 'recheck experimental' [15:45:39] hashar: I don't know if you heard, but swiftmailer is unlikely to end up being used now. Tim didn't think the features were worth the complexity. [15:45:59] ok [15:46:03] Tony's project is going ahead with a more direct implementation [15:46:11] my work is still valid for monolog and psr-X work isn't it ? [15:46:25] But hopefully monolog will pass muster :) [15:46:32] good ! [15:48:01] I have high hopes for more composer library usage in the future too. Probably starting with libraries extracted from our own codebase. [15:49:16] yup [16:17:47] 3Wikimedia Labs / 3deployment-prep (beta): Setup a Swift cluster to match production - 10https://bugzilla.wikimedia.org/62835#c1 (10Andrew Bogott) I'm going to try to set up a swift cluster in labs that beta can use. Will update as I go... Also, faidon just said "the other part that may need a modification... [16:30:37] Eeew. I keep hitting stupid bugs caused by the fact that the same table doesn't always have the same schema in every database. We have *got* to find a saner way to keep them in sync. [17:12:02] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c5 (10Andre Koopal) What are you exactly filling in? If I choose commons, and either leave the box before that empty or fill in commons as well, and choose some c... [18:59:22] 3Wikimedia Labs / 3deployment-prep (beta): Setup a Swift cluster to match production - 10https://bugzilla.wikimedia.org/62835 (10Greg Grossmeier) a:3Andrew Bogott [20:06:47] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c8 (10Matthew Flaschen) (In reply to Andrew Bogott from comment #6) > So the new puppet module installs virthosts and also removes any virthosts > that aren't currently instal... [21:19:47] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c9 (10Ori Livneh) (In reply to Matthew Flaschen from comment #8) > (In reply to Andrew Bogott from comment #6) > > So the new puppet module installs virthosts and also removes... [21:28:02] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c10 (10Tim Landscheidt) (In reply to Matthew Flaschen from comment #8) > [...] > But https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster says, > "This means tha... [21:30:32] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c11 (10Andrew Bogott) The fact that puppet actively destroys your local apache config is silly. If you want to puppetize and get me to merge your changes into the prod branch... [21:38:17] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c12 (10Ori Livneh) (In reply to Andrew Bogott from comment #11) > we definitely need a path to support local, stable changes that > don't require you to set up a puppetmaster.... [21:38:47] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c13 (10Ori Livneh) (These resources would be declared in a Labs-specific manifest.) [21:40:17] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c14 (10Andrew Bogott) Sure, that'll work. [21:41:18] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c15 (10Andrew Bogott) Well... it's moot now I guess, but I'd really prefer that puppet FAIL when there are untracked vhosts, rather than just cheerfully destroy them. [21:43:47] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c16 (10Ori Livneh) In fact, there's an even easier way to do this that doesn't require tampering with apache2.conf, and it's to replace the second (file_line) resource with:... [23:09:32] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~krinkle/MoreContributions/input.php - 10https://bugzilla.wikimedia.org/61036 (10Krinkle) 5ASSI>3RESO/WON [23:26:32] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~krinkle/MoreContributions/input.php - 10https://bugzilla.wikimedia.org/61036#c2 (10Krinkle) I've contacted Luxo to work with him on it. Can't promise how soon that'll happen though. MoreContributions will become a redirect to https:... [23:31:47] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c20 (10Andrew Bogott) ok -- now (well, in an hour or so) you should be able to recreate your sites by hand by placing the vhost files in /etc/apache2/sites-local. Please try...