[02:23:49] Analytics, Easy: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2704735 (Tbayer) >>! In T144714#2669554, @Tbayer wrote: ... > > This query does not work for several reasons. As I had said above, I unfortunately don't have time to help... [02:37:24] Analytics, Easy: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2704738 (Tbayer) PS: or even easier, use the PARSE_URL function described in the same section of the above linked documentation. [07:59:34] joal: o/ whenever you have 30 mins let's nuke aqs100[123] [08:09:21] Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2704949 (akosiaris) I 'll remove operations as I don't currently see how ops can help on this one. Also, I see no movement on this one, maybe it's resolved already ? [08:22:06] Hi elukey [08:22:29] !log Killing cassandra loading jobs for old aqs [08:23:30] elukey: Jobs killed [08:23:59] elukey: I'm going to update oozie prod code to be in sync with new-cassandra jobs [08:25:12] super! [09:15:53] mobrovac: hola! do you have a minute to chat about systemd::syslog? [09:17:34] oh right [09:17:51] i saw the patch, need to review it [09:17:56] gimme 5 mins and i'll do it [09:19:54] nono it is about another thing [09:19:59] namely rsyslog and firejail :) [09:20:32] I used systemd::syslog { 'pivot': but it doesn't work sadly since 'firejail' is the programname that rsyslog sees [09:21:28] so I was trying to figure out if there is an alternative of using systemd::syslog { 'firejail': [09:21:54] (that works fine, I tested manually, but it is not great) [09:32:05] oh wait [09:32:19] i think we have that working [09:32:22] since we run all services inside firejail [09:32:25] lemme take a look [09:33:58] elukey: https://github.com/wikimedia/operations-puppet/blob/production/modules/service/templates/initscripts/node.systemd.erb#L19 [09:34:26] just add that to the systemd unit file [09:34:27] ahhhhhhhhhhh [09:34:30] \o/ [09:34:40] going to add a note in the docs [09:34:43] thanks! [09:34:45] np [09:42:07] thanks for the review mobrovac :) [10:42:21] Analytics-Kanban, Patch-For-Review: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2705206 (elukey) Pivot has been deployed via Scap to stat1001 and now it is serving pivot.wikimedia.org. I updated https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Druid with basic admin info for... [10:44:24] \o/ --^ [10:44:34] except the visualization bug, pivot is running in prod :) [10:50:26] joal: going afk for lunch and then I'll start the procedure to nuke aqs100[123] [10:50:43] all good from your side? [10:50:43] Cool elukey [10:50:49] So far, no issue :) [10:51:01] super, will ping you again before starting :) [10:51:09] np elukey [11:09:16] ouch data loss error from oozie :( [11:13:55] seems "only" 9% [11:15:48] elukey: I'm about to start an oozie coord for it - Shall I or do you want to do it? [11:17:11] !log started 0026094-160922102909979-oozie-oozi-C to fix webrequest-load-check_sequence_statistics-wf-upload-2016-10-11-9 (oozie data consistency errors) [11:17:20] :) [11:17:24] Ok, this answers my question :) [11:17:28] Thanks elukey :) [11:47:44] joal: starting the nuke :) [11:47:53] NUKIIIIIIING ! [11:51:39] Analytics-Kanban: Update code in refinery-source and refinery to remove old-aqs compatibility - https://phabricator.wikimedia.org/T147841#2705311 (JAllemandou) [11:51:49] Analytics-Kanban: Update code in refinery-source and refinery to remove old-aqs compatibility - https://phabricator.wikimedia.org/T147841#2705323 (JAllemandou) a:JAllemandou [11:53:08] Analytics-Kanban: Update code in refinery-source and refinery to remove old-aqs compatibility - https://phabricator.wikimedia.org/T147841#2705311 (JAllemandou) [11:53:10] Analytics-Kanban: Decomission hosts from old aqs cluster - https://phabricator.wikimedia.org/T147460#2705331 (JAllemandou) [11:53:45] (PS4) Joal: Update casssandra loading classes for new AQS [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295663 (https://phabricator.wikimedia.org/T147841) [11:55:13] joal: ran puppet on aqs100[456] and all went fine (only firewall changes as expected) [11:55:28] I am going to wait a bit and then run puppet on aqs100[123] for the final strike [11:55:43] ok [11:55:58] elukey: checking graphs in the meantime [12:03:35] Analytics-Kanban: Migrate the simplest limn dashboards to dashiki tabular {frog} - https://phabricator.wikimedia.org/T126358#2705343 (mforns) Hi @HJiang-WMF! I'm migrating flow-reportcard and glam-metrics to Dashiki right now, and I saw in the task description that you are doing the same with ee-dashboard. C... [12:09:25] (PS1) Joal: Update cassandra load jobs to match new aqs needs [analytics/refinery] - https://gerrit.wikimedia.org/r/315241 (https://phabricator.wikimedia.org/T147841) [12:10:06] (CR) Joal: [C: -1] "Waiting for java code to be deployed." [analytics/refinery] - https://gerrit.wikimedia.org/r/315241 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [12:12:33] git up [12:12:36] oops :) [12:12:51] elukey: Added you to 2 reviews for new-aqs loading code in prod [12:13:18] (PS2) Joal: Update cassandra load jobs to match new aqs needs [analytics/refinery] - https://gerrit.wikimedia.org/r/315241 (https://phabricator.wikimedia.org/T147841) [12:13:40] joal: super :) [12:13:58] if you don't mind I'd proceed with the final strike [12:14:27] elukey: No issue on my side so far, please go ahead [12:16:04] !log Deploying refinery [12:16:44] elukey: Do you know if deployment.eqiad.wmnet has changed yesterday or today? [12:16:49] or maybe moritzm --^ [12:17:18] joal: it did, see mail to ops@ from 1-2 hours ago, we moved it to mira.codfw.wmnet in preparation of the tin update to jessie [12:17:38] moritzm: Didn't see the email, sorry - Thanks for the update ! [12:17:57] ping me or hashar if you notice any problems [12:18:14] moritzm: deploying from the new machione, I'll let you know [12:19:15] moritzm: failure for me :( Agent admitted failure to sign using the key [12:20:32] can you make a paste of the full error? [12:22:11] moritzm: https://gist.github.com/jobar/d34b78bd0630996f962864a9de7eac01 [12:22:23] moritzm: this is what scap deploy-log tells me [12:22:47] (CR) Elukey: Update casssandra loading classes for new AQS (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295663 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [12:23:37] (CR) Elukey: "Not in here, just realized that :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295663 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [12:24:03] PROBLEM - cassandra CQL 10.64.0.123:9042 on aqs1001 is CRITICAL: Connection refused [12:24:10] elukey: YAY ! [12:24:18] elukey: errors from the nuked machines :) [12:24:23] elukey: Nuke happened [12:25:06] (CR) Elukey: "LGTM but my expertise in oozie coordinators is limited :)" [analytics/refinery] - https://gerrit.wikimedia.org/r/315241 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [12:25:12] checking with hashar [12:25:21] sure moritzm [12:25:28] elukey: in case you have an idea --^ [12:25:35] o/ [12:25:42] elukey: I had a key error when trying to deploy from mira [12:25:46] Agent admitted failure to sign using the key. [12:25:52] hi hashar, thanks for having a look [12:25:53] that indicates an issue with the keyholder [12:25:58] that got the ssh keys in memory [12:26:03] hashar: I would have guessed so [12:26:25] most probably we would want to reload them [12:26:26] Analytics, Easy: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2705394 (Larsnooden) Thanks. (I don't suppose there is a test database with sample data to develop queries against? I know some of the expensive subscription databases... [12:27:20] moritzm: on mira would you mind rearming the keyholder? Should be: [12:27:22] sudo /usr/local/sbin/keyholder stop [12:27:25] sudo /usr/local/sbin/keyholder start [12:27:28] sudo /usr/local/sbin/keyholder arm [12:28:09] sure, give me 5-10 mins, currently looking at changeprop with Marko [12:28:34] hashar: I can do it [12:28:51] elukey: sure :) [12:29:11] elukey: https://wikitech.wikimedia.org/wiki/Keyholder has all the references to pwstore entries [12:29:20] yep yep page already opened [12:29:25] why the stop/start? [12:29:52] as opposed to directly: sudo /usr/local/sbin/keyholder arm [12:29:58] hashar: --^ [12:30:24] hmm [12:30:27] yeah just try arm [12:30:29] maybe that is enough [12:30:35] on beta I just reboot / restart evertything :D [12:30:45] cause I am too lazy to figure out the proper chain of action [12:31:28] PROBLEM - cassandra CQL 10.64.32.175:9042 on aqs1002 is CRITICAL: Connection refused [12:32:08] Received disconnect from 10.64.5.102: 2: Too many authentication failures for analytics [12:32:09] bah [12:32:50] ?? [12:32:56] was from tin [12:33:00] ahh okok [12:33:02] anyway [12:33:09] to repro scap behavior on that analytics repo [12:33:17] I think it ends up doing: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh analytics@stat1002.eqiad.wmnet [12:33:27] PROBLEM - cassandra CQL 10.64.48.117:9042 on aqs1003 is CRITICAL: Connection refused [12:34:43] hashar: keys armed [12:35:47] no luck :( [12:36:25] hashar: == not working? [12:40:01] yeah that is the same issue :( [12:40:04] try stop and start ? [12:42:48] hashar: better now? [12:42:51] I restarted keyholder-proxy [12:43:08] then [12:43:28] ohh hueehaze [12:43:47] might that i am in the proper unix group that can uses the key [12:44:04] you want me to try? [12:44:41] yes [12:45:31] I believe that is the unix group deploy-aqs [12:45:37] which you should all be member of [12:45:42] hashar: not for this one [12:45:51] hashar: analytics for the deploy I try [12:46:04] hashar: success :) [12:46:09] thanks hashar and elukey :) [12:46:11] elukey: you fixed it ! :] [12:46:24] \o/ [12:47:03] we have switched to mira [12:47:18] and I guess some corner case thing was not in sync with whatever was on tin [13:00:06] !log Start prod version of wdqs_extract job [13:44:29] !log merged https://gerrit.wikimedia.org/r/315101 on stat1002 (removal of ::statistics:wikistats) [13:44:52] this should allows us to remove an old cron and some perl package [13:54:59] elukey: ciao o/ [13:55:35] hola! [13:55:57] como estas? [13:56:24] * urandom tries to figure out how to do an upside down question mark... [13:57:34] ¿cómo estás? [13:57:36] :) [13:57:55] todo bien :) [13:59:29] urandom: all good from your side? [13:59:37] ya! [13:59:56] coming off a 3-day weekend [14:00:39] yesterday was indigenous peoples day (or Columbus Day if you are a traditionalist) [14:01:13] but since columbus isn't exactly someone you want to celebrate, i am not a traditionalist :) [14:01:48] :D [14:02:02] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2705618 (Ottomata) > In any case, browser use cases will have to download a client implementation, while with SSE most users will already have that implement... [14:02:59] urandom: I am reviewing https://gerrit.wikimedia.org/r/#/c/314603, but if you could give me a quick summary it would be great [14:04:19] elukey: right, well, other (already merged) gerrits resulted in the jar being deployed to all RESTBase hosts (https://gerrit.wikimedia.org/r/313825, https://gerrit.wikimedia.org/r/313892) [14:04:37] and of course https://gerrit.wikimedia.org/r/313619 made the classpath configurable [14:04:58] "the jar" here being the TimeWindowCompactionStrategy jar [14:05:20] yep yep.. I checked on restbase1010 and aqs1004 (after https://puppet-compiler.wmflabs.org/4288/), it looks good [14:05:31] so this change will only go to restbase* hosts [14:05:59] right, and technically, merging this will do nothing [14:06:15] even after a restart of Cassandra, it will only add the jar to the classpath [14:06:38] we will then need to ALTER KEYSPACE to enable it [14:06:51] which we'll do on a low-to-middle traffic keyspace [14:07:10] we've been testing it in Staging (for whatever that is worth), for a couple of weeks now [14:07:11] it looks very good tom e [14:07:16] ready to merge? [14:07:30] yup! [14:08:33] (PS1) Milimetric: Allow dashboard to use wiki-specified defaults [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/315267 [14:37:48] milimetric, hi1 yt? [14:37:53] hi!* [14:39:18] hi :) yes, in cave tlkin to andrew [15:00:37] Analytics-Kanban: Kill limn1 - https://phabricator.wikimedia.org/T146308#2705795 (Nuria) [15:00:39] Analytics-Kanban, Patch-For-Review: Kill dashboards on limn1 that are no longer used - https://phabricator.wikimedia.org/T147000#2705794 (Nuria) Open>Resolved [15:00:56] Analytics-Dashiki, Analytics-Kanban, Need-volunteer, Patch-For-Review: Vital-signs layout is broken - https://phabricator.wikimedia.org/T118846#2705796 (Nuria) Open>Resolved [15:01:26] ottomata, joal: standduppp [15:08:31] AKKKKK [15:08:32] SOERY [15:30:51] (PS1) Elukey: Update per-article compression scheme to default (LCS) [analytics/aqs] - https://gerrit.wikimedia.org/r/315283 (https://phabricator.wikimedia.org/T140866) [15:30:55] (PS1) Elukey: Map null values to zeros in output in per-article endpoint [analytics/aqs] - https://gerrit.wikimedia.org/r/315284 (https://phabricator.wikimedia.org/T144521) [15:41:41] (CR) Nuria: [C: 2 V: 2] Map null values to zeros in output in per-article endpoint [analytics/aqs] - https://gerrit.wikimedia.org/r/315284 (https://phabricator.wikimedia.org/T144521) (owner: Elukey) [15:42:00] (CR) Nuria: [C: 2 V: 2] Update per-article compression scheme to default (LCS) [analytics/aqs] - https://gerrit.wikimedia.org/r/315283 (https://phabricator.wikimedia.org/T140866) (owner: Elukey) [15:42:23] (Abandoned) Nuria: Updating master with new-aqs-cluster branch [analytics/aqs] - https://gerrit.wikimedia.org/r/314722 (https://phabricator.wikimedia.org/T144497) (owner: Nuria) [15:44:31] Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2705924 (Nuria) Open>Resolved [15:51:55] (CR) Nuria: Update casssandra loading classes for new AQS (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295663 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [15:53:15] (CR) Nuria: [C: 2 V: 2] "Thanks for doing this, much better now." (1 comment) [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/315267 (owner: Milimetric) [15:56:37] Analytics, Easy: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2705947 (Nuria) >(I don't suppose there is a test database with sample data to develop queries against? I know some of the expensive subscription databases used to offer tr... [15:58:08] milimetric: are you ok? [16:27:34] joal: you there? [16:27:52] elukey: yop [16:28:01] do you have a minute for a druid question? [16:28:06] elukey: I do ! [16:28:11] thanksss! [16:28:16] elukey: batcave? or IRC? [16:28:23] IRC is fine [16:28:28] k [16:29:15] so pivot seems to create the datacubes using "title" and "description", meanwhile it is not using "name" (e.g. "pageviews-daily", etc..) [16:29:26] Ahhhhh :) [16:29:39] we can specify them via config file [16:29:39] BUT [16:29:50] elukey: I think we didn't investigate further naming than the datasource name :) [16:29:53] it would be awesome to have them auto-populated via druid [16:30:06] elukey: give me aminute: ) [16:30:11] \o/ [16:33:37] elukey: looking at http://druid.io/docs/latest/querying/datasource.html and the coordinator UI, I don't find any easy way to populate such a name field :( [16:34:40] joal: I am pretty sure that adding "description" and "title" will do the magic [16:34:53] elukey: ok, but where? [16:41:42] elukey: The only metdata I can get from druid are time oriented (max ingested timestamp) [16:43:24] joal: ah sorry I thought that the json snippets where related to how configure the tables [16:43:40] elukey: might [16:44:48] my 2 minutes reading on pivot makes me think the datasource config is to be manual :( [16:45:03] elukey: but it;s a 2 minutes reading [16:46:12] okok I'll do some investigation tomorrow, thanks :) [16:46:48] elukey: alos reading again in schemas provided at load time, looks like title and desc are not fields for druid (must be pivot only) [16:46:51] elukey: sorry :( [16:47:51] nono good news, I know what to do :) [16:47:59] :) [17:05:00] ottomata: in here is quiet :) - so I completed all the steps but the partition creation one [17:05:15] I umounted /var/spool/kafka/i [17:05:22] and now I can't see any disk [17:05:28] even with fdisk [17:08:27] on phone! brt, and in ops meeting, in a sec [17:10:02] ah ok so I'll try to experiment :) [17:12:34] ok elukey, in meeting now, but am playing too [17:13:06] hm [17:13:07] fdisk: cannot open /dev/sdi: No such device or address [17:13:15] elukey: maybe we should just reboot the node [17:15:13] yeah I was thinking the same [17:15:33] kafka is systemctl masked [17:15:47] oh ok great [17:15:53] elukey: you doing it or shoudl I? [17:16:02] going to do it [17:16:06] k [17:21:00] wikimedia/mediawiki-extensions-EventLogging#606 (wmf/1.28.0-wmf.22 - 63b06af : Tyler Cipriani): The build has errored. [17:21:00] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/63b06af6f2bc [17:21:00] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/166819867 [17:24:21] ottomata: I am re-rebooting, it doesn't come up cleanly [17:26:57] hm [17:27:02] what's happening? [17:27:19] Welcome to emergGive root password for maintenance [17:27:21] :) [17:27:41] it froze when I tried the Control-D way (proceed anyway) [17:28:42] ottomata: ok I managed to log in as root and we can run fdisk from the mgmt console [17:36:51] ottomata: no same problem [17:41:58] hmmm [17:42:04] same problem with fdisk [17:42:06] ? [17:42:10] no such devie? [17:42:12] device? [17:42:38] no no I think I didn't create the partition correctly [17:42:42] I just did it [17:42:49] do you want to double check in the mgmt console? [17:45:25] ohh worked! [17:45:27] rebooting [17:45:30] (PS1) Mforns: [WIP] Add data cube title to config [analytics/pivot] - https://gerrit.wikimedia.org/r/315315 (https://phabricator.wikimedia.org/T138262) [17:47:31] Analytics-Kanban, Patch-For-Review: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2395083 (mforns) @elukey @Milimetric This ugly hack may solve the label problem, if nothing else does. I haven't tested yet, but I thought it would be cool to have it here for you guys to see. Feel free... [17:48:32] I'm off for tonight a-team, see you tomorrow [17:48:49] bye joal !! cya [17:49:16] mforns: I thank you for having shown me the way toward better testing [17:49:29] xDDDDD [17:49:41] mforns: It soooooo long for me to thorough enough, but bugs are killed on the way ;) [17:50:03] cool :D [17:53:37] so ottomata the disks seems mounting correctly but I can't manage to boot correctly [17:58:11] ah snap! Oct 11 17:57:04 kafka1018 mount[1225]: mount: /dev/sdc1 is already mounted ...sy [17:58:20] ottomata: I think we already encountered this issue [18:00:40] I think that the fstab is wrong [18:02:05] OH [18:02:13] elukey what is in fstab? [18:02:27] ottomata: batcave? [18:07:05] k [18:32:45] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [18:33:26] (CR) Nuria: Improve build a bit more (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/314622 (owner: Milimetric) [18:38:06] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [18:39:53] https://www.irccloud.com/pastebin/KS1R7tsS/ [18:40:05] mforns: how can i see errors in logstash again? the dashboard linked on docs doesn't have the info , does it? [18:44:29] mforns: got it, nm [18:44:45] oh nuria sorry didn't see the ping [18:49:45] nuria, I added the link to the dashboard in logstash's home now [18:50:04] in the top menu [18:50:23] mforns: k, thank you. just pinged mobile folks again [18:54:36] Analytics, Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2706774 (Ottomata) [18:57:10] !log kafka1018 back in service after maintenace [18:57:18] *maintenance [18:57:23] all right logging off people! [18:57:26] will double check later [19:00:24] Analytics-Kanban, Datasets-Webstatscollector, RESTBase-Cassandra, Patch-For-Review: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2706831 (Nuria) [19:00:26] Analytics-Kanban, Patch-For-Review: Clean up references on puppet code to old AQS cluster - https://phabricator.wikimedia.org/T147461#2706830 (Nuria) Open>Resolved [19:01:50] * ottomata -> cafe [19:03:18] Analytics, Analytics-Dashiki: Migrate from bower to yarn or just npm - https://phabricator.wikimedia.org/T147884#2706880 (Milimetric) [19:14:19] hey all, I'm on my phone internet, I think my normal internet will be down for a while and I can't burn too much of my data plan [19:14:29] I'll try to find an alternative, but for today it's ok [19:14:40] mforns: when you working 'till? [19:21:37] msg milimetric: as you wish but you know, taking the day off is also an option [19:22:52] thx :) it's ok, I thought about it but I'd just be bored. I'll deal with the bad connection and try to get some work done [20:04:23] Analytics-Kanban: Spamy - User-like pages distort our pageview metrics (they return 200 when they should return 404) - https://phabricator.wikimedia.org/T145922#2707100 (Nuria) To wrap this up here is an example of pageviews that we are not serving. Numbers for pageviews like 'GoogleAnalitycsRoman/some" in S... [20:04:32] Analytics-Kanban: Spamy - User-like pages distort our pageview metrics (they return 200 when they should return 404) - https://phabricator.wikimedia.org/T145922#2707102 (Nuria) Open>Resolved [21:12:11] (PS1) Aklapper: Replace Bugzilla links by Phabricator links [analytics/wikistats] - https://gerrit.wikimedia.org/r/315417 [21:12:46] (Abandoned) Aklapper: Replace Bugzilla links by Phabricator links [analytics/wikistats] - https://gerrit.wikimedia.org/r/281112 (owner: Aklapper) [21:31:15] (CR) Milimetric: [WIP] Add data cube title to config (1 comment) [analytics/pivot] - https://gerrit.wikimedia.org/r/315315 (https://phabricator.wikimedia.org/T138262) (owner: Mforns) [21:32:55] Analytics-Kanban, Patch-For-Review: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2707483 (Milimetric) it's not ugly, and should probably be upstreamed anyway if they haven't fixed it otherwise. I left a suggestion on the patch and we should totally do this if there's nothing easier... [21:43:03] Analytics: Puppetize job that saves old versions of geoIP database - https://phabricator.wikimedia.org/T136732#2707531 (Milimetric) In my opinion geowiki is a very fragile process and I like having this backup. Killing the cron would not make me feel safe. Fixing geowiki to something more reliable and moni... [21:44:35] Analytics: Pre-generate mysql ORM code for sqoop - https://phabricator.wikimedia.org/T143119#2707532 (Milimetric) Oh yeah, it's relevant even if we run sqoop once, because for every table in every database it repeats the column detection process (so like 5000 times every run). I should have probably not ski... [21:46:16] Analytics-Kanban: Improve mediawiki data redaction - https://phabricator.wikimedia.org/T146444#2707538 (Milimetric)