[18:18:08] o/ YuviPanda [18:18:25] Interested in being on standby while I push to prod today? [19:24:38] o/ Krinkle [19:24:47] Can you help me translate "Edit quality (20k random sample, 2015)" to dutch? [19:25:12] I'm going to set up a labeling campaign for nlwiki and I figure you might do better than gtranslate [20:04:55] halfak: I'm here if you wanna deploy [20:05:14] Cool. I'll get started. [20:06:18] halfak: ok! [20:06:31] * YuviPanda stands by and writes up interview notes [20:12:22] YuviPanda, so, those updates to puppet happened. Should I just assume that those packages have been installed. [20:12:29] I suppose I can check. [20:12:35] halfak: yes [20:12:38] halfak: so it runs every 20min [20:12:45] you can check too! :D [20:12:58] How do you check if apt installed a package? [20:13:02] Is there an ls for apt? [20:13:57] Meh. Just tried to install one of the packages again and it was already there [20:14:00] good enough :) [20:14:21] halfak: so I know there's apt-policy or something but what you did is exactly what I do :) [20:14:40] :D [20:14:53] oooo [20:16:00] * halfak starts the celery deploy [20:16:23] * halfak watches the model files download [20:16:30] One down [20:17:56] All the workers seem to be online and taking stasks still. [20:17:59] That's a good sign. [20:18:13] * halfak starts web deploy [20:18:49] Now the loooong wait for the uwsgi restart [20:19:01] yeah halfak can you file a bug about that and tag it as operations? [20:19:10] I can convince ori or godog to dig into that [20:19:21] since I noticed it's happening to other uwsgi servers too [20:19:22] and not just ORES [20:19:41] OK will do [20:19:58] halfak: thanks! also stern words on the backup tiket :) [20:20:46] Might need you to help me fill in details. Will make the ticket now though. [20:21:52] halfak: sure! mostly a 'dammit I need this and because this does not exist I am doing terrible things' [20:22:02] https://phabricator.wikimedia.org/T118495?workflow=create [20:22:13] Oh I'll do details on the backup rant. [20:22:20] I just need your details on the uwsgi slow :D [20:23:06] :D [20:23:08] yeah I'll do that [20:24:27] And we're done [20:24:31] looks like it is all working. [20:24:36] I'll restart the precacher [20:25:23] Woops. I broke graphite. Fixing. [20:25:56] :D [20:25:59] <3 [20:27:42] * halfak pushes the fix out to staging [20:28:32] Waiting for that uwsgi restart [20:29:08] heh [20:29:17] halfak: we should make a dashboard of pretty graphs at some point [20:29:32] Can we hit graphite with an API? [20:29:53] we can do better [20:29:56] https://grafana.wikimedia.org/dashboard/db/varnish-http-errors [20:30:00] so that's an example [20:30:10] that's all coming from graphite [20:30:10] oooo [20:30:40] halfak: :D [20:30:54] How would you track queue size? [20:31:00] incr seems wrong [20:31:02] timing too [20:31:05] guage [20:31:09] ahh [20:31:12] it's an absolute value [20:32:09] OK. Staging looks good. On to prod! [20:34:10] Say, we need to push our "worker" count on the web workers. [20:34:33] I'll go do that an wikitech before I deploy to web [20:35:28] halfak: yeah you can do that in Hiera:Ores for it to take effect everywhere [20:35:34] halfak: and then 20min wait or run 'sudo puppet agent -tv' [20:35:37] Cool Was just going to ask [20:35:59] I'll sudo puppet so that we can do one restart during the deploy and be done. [20:36:20] halfak: kkk [20:36:25] NO [20:36:26] :P [20:36:29] Too many k's [20:36:32] 4 is OK [20:36:34] 2 is OK [20:36:35] 1 is fine [20:36:39] BUT NOT THREE [20:36:46] halfak: although I fixed the bug about worker count changing not restarting uwsgi [20:36:57] halfak: is it ok if they're all Kapital? [20:37:00] Oh gotcha. So it doesn't matter then. [20:37:02] Worse [20:37:48] Oh man. So puppet is going to do it's *own* restart!? [20:37:54] Oh... No... it didn't [20:38:12] halfak: is it dropping requests during the restart? [20:38:19] halfak: did it change the worker count? [20:38:47] Not sure about the dropped requests. [20:38:52] But I don't see a change in worker count yet [20:39:02] right so when that changes it'll restart [20:39:11] But the config is changed. [20:39:15] oh [20:39:22] Oh.. wait... no it isn [20:39:23] grr looks like I didn't fix the bug properly afterall >_> [20:39:25] 't [20:39:26] ah [20:39:28] ok [20:39:30] :D [20:39:51] so it's up in the air if I fixed the bug or not [20:39:53] How many CPUs on our web nodes? [20:40:08] uh [20:40:12] cat /proc/cpuinfo? [20:40:27] halfak: don't fully remember... [20:40:37] 2! [20:40:39] That's why [20:40:42] ah [20:40:44] :) [20:40:59] Our staging box has 4! [20:41:05] Anyway, this is better. [20:41:06] right because it's also running celery [20:42:18] OK. All good now. [20:42:38] "Loading facts" [20:43:09] Here we go with the web deploy again [20:43:10] :) [20:51:11] YuviPanda, looks like we're done! [20:51:28] halfak: \o/ [20:51:32] yay [20:52:41] halfak: http://jvns.ca/blog/2015/11/09/docker-is-amazing/ is someone who is describing why they found docker useful [20:52:50] (and probably very close to the reason for having a Dockerfile in revscoring) [20:54:51] Yeah. This makes sense. I need to figure out a good pattern and write a step-by-step for our next hackathon. [20:55:32] Speaking of which, I'm looking into reinstalling my host. Might do that tonight. I'm also past the age where installing an OS is exciting. [20:55:52] But I want to move past 14.04 to see if I can get some better 2d acceleration. [20:56:41] "Ubuntu 15.10 comes with nine months of security and maintenance updates." 9 months!? [20:56:47] I thought you'd at least get a year [20:56:50] * halfak gumbles [20:57:28] halfak: heh :) [20:57:36] * YuviPanda is on debian these days but ubuntu too [23:59:10] Hey YuviPanda. Can I ask you to be present briefly again? It turns out that our last deployment was very lucky. [23:59:24] I want to try a new push to prod. [23:59:45] The requirements did not propagate. So we're running on some old code with old behavior.