[06:04:57] joal: o/ [06:05:00] morninggggg [06:09:31] if you are ok I'd proceed with https://gerrit.wikimedia.org/r/#/c/429429/ (applying CMS to the namenodes) [07:21:30] Hi elukey - Let's go for it! [07:23:01] \o/ [07:23:06] just tested it in labs, all good [07:23:52] so plan is: merge, restart namenode on an1002, check metrics for a couple of hours, do the same with an1001 [07:30:36] ok merged, now restarting namenode on an1002 [07:33:41] restarted [07:33:51] watching [07:35:36] elukey: we haz new metrics :) [07:37:09] elukey: tis patch doesn't bump the heap, right? [07:38:29] elukey: sonething weird as well: number of corrupted blocks [07:39:06] joal: nope I added only the new settings as suggested by cloudera [07:39:16] the corrupted block thing might be a weirdness in jmx [07:39:28] k [07:40:03] I need to check if the metric is a total counter or not, but I suspect that sometimes it is not updated [07:40:28] because I rememeber a lot of times in which me and you did the check on hdfs via cli and nothing popped up [08:07:12] elukey: new GC time is not looking great, he? [08:07:37] I was about to write someting :) [08:07:41] :) [08:07:57] elukey: gave some time to stabilize, but man, that's not what we would expected I assume [08:08:34] so GC Time seems to be more constant, around 6/7s for old gen, that is good afaics (I am monitoring also the JvmPauseMonitor occurrences) [08:08:49] what we'd need to avoid is those spikes to 25s that kills us [08:08:51] elukey: Hahhh [08:09:05] elukey: you could reimage a couple of workers :) [08:09:09] ahhahah [08:09:23] GC runs are definitely more [08:09:45] but given the nature of CMS it might be a consequence of how it behaves [08:10:22] elukey: other interesting thing I have noticed: https://grafana-admin.wikimedia.org/dashboard/file/server-board.json?panelId=7&fullscreen&orgId=1&var-server=analytics1002&var-network=eth0&from=now-1h&to=now&refresh=1m [08:10:54] yes that one was expected, CMS grabs some threads to run its logic [08:11:17] elukey: from almost nothing to ~8% CPU usage, that's a lot ! [08:13:20] joal: yeah but I'd say that we can afford to sustain 8% of cpu utilization :D [08:13:28] agreed :) [08:14:08] one thing that we could do is to force a failover to analytics1002 [08:14:15] and see how timings change [08:14:55] elukey: +1 [08:15:29] all right doing it [08:17:12] joal: done! [08:17:57] ah while we wait for the metrics [08:18:22] do you remember that a while ago we were wondering how frequent the namenode takes snapshots of its metadata? [08:18:29] I think that we have the default, 1h [08:18:53] elukey: Interesting! [08:19:12] elukey: You confirm the passive namenode does that, not the active one, right? [08:19:24] yep it seems so [08:19:24] -rw-r--r-- 1 hdfs hdfs 1806544161 Apr 30 06:28 fsimage_0000000001894266265 [08:19:28] -rw-r--r-- 1 hdfs hdfs 62 Apr 30 06:28 fsimage_0000000001894266265.md5 [08:19:31] -rw-r--r-- 1 hdfs hdfs 1807687684 Apr 30 07:28 fsimage_0000000001894538880 [08:19:34] -rw-r--r-- 1 hdfs hdfs 62 Apr 30 07:28 fsimage_0000000001894538880.md5 [08:19:45] and we keep the last two snapshots afaics [08:20:04] Interesting! [08:20:09] Thanks for letting me know:) [08:20:23] thanks for letting me know how it worked! :D [08:21:04] elukey: we are like physicians - I study theory and models, you work ground battles :) [08:22:18] ahahah [08:25:35] joal: I am watching https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=26&fullscreen&orgId=1&from=now-1h&to=now, selecting only an1001/2 old gen metrics [08:26:10] I was concentraing on this as well [08:26:16] The pattern is very interesting [08:26:44] and also no org.apache.hadoop.util.JvmPauseMonitor occurrence during the past hour [08:26:55] (there was one every 10 mins more or less) [08:27:12] THAT is super awesome :) [08:27:23] ( grep org.apache.hadoop.util.JvmPauseMonitor /var/log/hadoop-hdfs/hadoop-hdfs-namenode-analytics1002.log) [08:27:35] elukey: look that drop in old GC time for an1001 [08:27:57] I think those high time are due to first-snashot creation [08:28:26] might be yes [08:29:06] I might be too early but I am happy about CMS [08:29:16] Another interesting finding: only diff between active/passive modes is disk usage :) [08:29:25] elukey: goooood :) [08:29:35] * joal is happy when elukey is happy :) [08:30:06] https://www.youtube.com/watch?v=d-diB65scQU [08:30:55] * elukey sings with Joseph [08:37:29] joal: I'd say we restart namenode on an1001, wait a bit and then failover again [08:37:38] works for me elukey :) [08:37:47] ack! [08:41:09] restarted [08:43:24] elukey: the pattern match between moving an1002 and an1001 to new GCs is kinda freaky [08:44:51] joal: what do you mean? [08:45:05] https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=26&fullscreen&orgId=1&from=1525073612898&to=1525077895920 [08:45:15] elukey: beginning and end of chart [08:46:14] ah yes the restart is always a bit intensive on the namenode [08:47:01] joal: take also a look to [08:47:01] https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=58&fullscreen&orgId=1&from=now-3h&to=now [08:47:31] elukey: nono - I mean, the new GC times between an1002 and an1001 - They are EXACTLY the same !! [08:48:00] ah yes! [08:48:03] aahhahah [08:48:25] Not true for newGen, but for old-gen, man, that is almost scary !! [08:48:26] :D [08:50:48] (03CR) 10Joal: "Done nuria" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429380 (owner: 10Joal) [08:55:48] it is weird that an1001's GC timings have not yet recovered [08:56:24] hm [08:58:04] this is the problem of having too many metrics [08:58:05] ahahhah [09:00:06] elukey: Would you mind telling when was the previous snapshot from an1001?> [09:02:40] root@analytics1001:/home/elukey# ls -lht /var/lib/hadoop/name/current/fs* [09:02:43] -rw-r--r-- 1 hdfs hdfs 62 Apr 30 07:29 /var/lib/hadoop/name/current/fsimage_0000000001894538880.md5 [09:02:46] -rw-r--r-- 1 hdfs hdfs 1.7G Apr 30 07:29 /var/lib/hadoop/name/current/fsimage_0000000001894538880 [09:02:49] -rw-r--r-- 1 hdfs hdfs 62 Apr 30 06:29 /var/lib/hadoop/name/current/fsimage_0000000001894266265.md5 [09:02:52] -rw-r--r-- 1 hdfs hdfs 1.7G Apr 30 06:28 /var/lib/hadoop/name/current/fsimage_0000000001894266265 [09:02:55] its 9:02 UTC now [09:03:26] aiement Par Carte [09:03:32] wow - sorry about that [09:03:35] hahahaahh [09:03:37] hm [09:05:25] from the logs it seems doing its work [09:06:12] well, let's leave it do then :) [09:09:23] it would be nice to know though what the hell it is doing, that horrible GC timing is not good [09:12:51] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Operations, and 2 others: Kafka API negotiation errors on kafka main brokers - https://phabricator.wikimedia.org/T193238#4167452 (10mobrovac) [09:19:37] the only thing that I keep seeing is [09:19:37] 2018-04-30 09:19:23,431 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error [09:19:43] for various users [09:23:38] (that should be ok) [09:23:44] trying jconsole [09:27:27] so the oldgen seems 4.2G and completely full [09:27:43] so it might be in a GC loop since it keeps trying to clean up memory [09:28:21] elukey: the patterns in GC time is as if it was trying again and again to do the same thing [09:31:02] so from jconsole it is telling me that the GC time is 10s [09:31:16] possibly the graph is not plotting the right value [09:31:18] increase(jvm_gc_collection_seconds_sum{instance=~"analytics100[12]:10080"}[5m]) [09:31:31] I now remember that I had to tune this [09:32:21] so basically, since we get a counter of total GC time spent, I added a measure about delta for the lowest time window, 5m [09:32:48] in any case, Old gen is full [09:32:53] lemme check how it is for an1002 [09:33:34] we could instead plot the rate per second of gc time [09:33:40] that might be more accurate [09:34:08] old gen heap full even with an1002 [09:34:10] mmmm [09:34:35] rate for jvm_gc_collection_seconds_sum gives me 300ms :( [09:35:12] yeah that will be time spent per second in GC [09:38:33] now I am wondering i f XX:CMSInitiatingOccupancyFraction=70 is not ok for our use case [09:38:58] elukey: and/or - Maybe 6Gb for a NN is too small? [09:39:49] might be.. so I tried to restart the namenode on 1001 again to see if it improves, if not I'd proceed with removing CMSInitiatingOccupancyFraction [09:40:00] k [09:43:31] joal: another thing would be that CMSInitiatingOccupancyFraction=70 + a heap that is not super big (like you were saying) might trigger these weird things [09:44:09] elukey: That's my idea as well (small heap + try to keep empty = really small portion of space to actually work!) [09:44:28] yeah [09:44:41] elukey: now, why did it work with an1002???? [09:45:13] it might be just luck, with an1001 we might have crossed the threshold that triggers the continous GC [09:45:25] (speculations) [09:47:33] so now timings are good [09:47:47] Man - That's weird [09:48:13] I am pretty sure it is that CMSInitiatingOccupancyFraction=70 [09:48:37] elukey: I'd give it more RAM instead of removing tat param, don't you htink? [09:49:07] joal: I was debating with myself that same thing [09:49:09] 8g ? [09:49:10] :D [09:49:22] yeah, let's start with that [09:50:30] done [09:54:34] elukey: I like that pattern of GC counts now :) [09:55:07] elukey: let's wait before assuming it's good, but at least the first glimpse of it makes me fel good [09:55:27] joal: we were happy before this morning :P [09:55:39] elukey: I changed on purpose :) [09:59:25] elukey: would you mind changing the heap size for an1002, swapping active and restart? [09:59:32] joal: what do you think aobut doing a failover? [09:59:34] ahahhahah [09:59:41] yes sure :) [09:59:43] I'm eager to see those GC counts reduce :) [10:00:01] I have already puppetized the GC increase [10:00:06] awesome [10:00:27] an1001 is again the active NN [10:03:14] elukey: an1001 out of ongoing GC loops I hink :) [10:03:29] elukey: CPU drop :) Hoorayn [10:04:58] \o/ [10:05:00] elukey: on becoming active, a bit more of NewGen GC, but no bump in olgGen - Gooood [10:05:03] going to restart 1002 in a bit [10:05:32] so another learning today is that NN is really sensitive when we talk about heap space [10:05:40] elukey: yes, let's releawse this poor CPU of trying to empty a room too small for the thing tio put in it :) [10:06:10] elukey: not sure I'd put it the way you do - I'd say: NN needs RAM, let's make sure we give ti some [10:06:52] joal: yeah, but it worries me a bit to have a dependency between number of files stored and heap space :D [10:07:00] elukey: why? [10:07:28] elukey: The purpose of NN being to manage those files, i actaully makes sense doesn't it? [10:07:50] because there is no free lunch in my opinion, when the heap grows it also means that other things like GC etc.. might have more challenges in doing their work under pressure [10:08:01] it makes sense indeed, it worries me as ops :D [10:08:04] agreed [10:08:41] If you want free lunch, please feel free to come home anytime - But it's surely not hadoop-realted ;) [10:10:13] elukey: an1002 out of CPU4GC loop :) [10:10:29] elukey: Thanks mate for fine tuning!h [10:10:43] \o/ [10:14:58] elukey: now I'll be curious for reall about what happens when you reimage hosts :) [10:17:36] will do it this afternoon! [10:17:56] I'd say that we could postpone druid's migration to wednesday, to avoid too many inflight things [10:18:08] elukey: as you wish :) [10:20:18] elukey: it's also interesting to notice that, before we started to work on NN, heap was 6G and the thing was working, with used-heap ~ max-heap [10:20:50] elukey: The GC change says: Let's make sure we try to have heap no more than 70% full [10:22:06] elukey: What makes he thing happy is obviously to move max-heap to 6G (previsouly used heap) / 0.7 (70 percent) ~ 8.5G :) [10:24:48] :) [10:37:40] joal: going to lunch + errand, I think that everything is stable now [10:37:53] thanks a lot! [10:38:05] I think the end result is a big win for us :) [10:38:18] elukey: thank YOU, as usual ;) [10:50:18] 10Analytics-Kanban, 10Analytics-Wikistats: Add druid datasources as configuration parameter in AQS - https://phabricator.wikimedia.org/T193387#4167623 (10JAllemandou) [10:50:21] (03PS1) 10Joal: Add a config param for druid datasources [analytics/aqs] - 10https://gerrit.wikimedia.org/r/429765 (https://phabricator.wikimedia.org/T193387) [11:11:54] 10Analytics-Kanban, 10Analytics-Wikistats: Index by-snapshot mediawiki-hsitory-reduced in druid - https://phabricator.wikimedia.org/T193388#4167661 (10JAllemandou) [11:12:40] (03PS3) 10Joal: Add optional datasource to druid loading workflow [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405053 [11:12:42] (03PS1) 10Joal: Add snapshot to datasource-name (mw hist reduced) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429770 (https://phabricator.wikimedia.org/T193388) [12:20:51] (03PS1) 10QChris: Add .gitreview [analytics/wmde/WDCM-Biases-Dashboard] - 10https://gerrit.wikimedia.org/r/429782 [12:20:54] (03CR) 10QChris: [V: 031 C: 032] Add .gitreview [analytics/wmde/WDCM-Biases-Dashboard] - 10https://gerrit.wikimedia.org/r/429782 (owner: 10QChris) [12:27:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update ua-parser package. Both uap-java and uap-core - https://phabricator.wikimedia.org/T192464#4167805 (10fdans) a:05fdans>03Nuria [12:31:59] * joal loves the effect of the new heap/gc conf on namenodes :) [12:58:03] it looks indeed really stable and efficient, gooooooood [13:03:21] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Hadoop HDFS Namenode shutdown on 26/04/2018 - https://phabricator.wikimedia.org/T193257#4167828 (10elukey) CMS settings alone were not sufficient to make everything work, when we restarted the namenode daemon on analytics1001 (that was the standby at the t... [13:05:43] 10Analytics: Varnishkafka does not play well with varnish 5.2 - https://phabricator.wikimedia.org/T177647#4167830 (10elukey) Thanks a lot for all the research work, I am following your progress and I'll try to dedicate some time in reviewing the code during this month! [13:07:54] joal: whenever you are back online, if you feel adventurous we can try to upgrade druid analytics :) [13:08:21] in the meantime, I am going to reimage two hadoop workers to see how it goes with the new GC settings [13:09:07] and the winners are.. 1050 and 1049 :) [13:11:09] hoi [13:11:34] ottomata: o/ [13:11:38] watch 'ps aux | grep yarn | grep -c -v grep' is really awesome [13:11:39] :) [13:14:03] :) [13:14:39] so it seems that we found a good GC setting for the namenode [13:15:24] but to make it work properly we had to go to Xmx/Xms 8G [13:15:48] heap consumption stayed around 6G as it was before, so I think we'll be fine for a while [13:18:38] ok great [13:18:39] sounds good [13:18:47] we can probably go more than 8G now if you think that would be wise [13:21:29] nono for the moment I think we are fine, but let's keep an eye on those GC metrics in the future [13:21:45] k [13:21:54] (fyi am beginning kafka200[23] reimage) [13:22:00] nice! [13:22:23] did you see https://phabricator.wikimedia.org/T193238 ? [13:22:46] now statsv has 0.9 hardcoded basically [13:23:17] 10Analytics: Varnishkafka does not play well with varnish 5.2 - https://phabricator.wikimedia.org/T177647#4167903 (10R4q3NWnUx2CEhVyr) Thanks, however we are hitting an issue where with varnish 5.2 we are getting Segfaults... We tried two paths: 1. porting the changes in VSL/VSM APIs 2. changing to VUT both get... [13:23:52] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4167904 (10Ottomata) [13:24:06] elukey: i did [13:24:16] sounds great, i'll add it to list of things to change for main upgrade [13:24:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4167905 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1049.eqiad.wmnet', 'an... [13:24:24] ack [13:25:43] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#4167912 (10Ottomata) [13:26:33] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4167916 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['ka... [13:29:49] ottomata: helloooo what do we do with EL's tests? [13:30:09] they are really _testing_ my patience :P [13:31:04] hmm, i think i saw someone submit something.. [13:31:17] fdans: https://gerrit.wikimedia.org/r/#/c/429651/ [13:31:27] for your patch we can skip jenkins [13:31:30] but we should get it fixed [13:31:32] maybe that patch will do it? [13:31:36] not sure, it got a -1 too [13:32:36] (03CR) 10Milimetric: [V: 032 C: 032] "nice" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429380 (owner: 10Joal) [13:34:32] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4167936 (10Ottomata) [13:35:50] ottomata: my patch also included that fix, now there's a different error when importing the test suite in setup.py [13:36:00] oh yours does the same? [13:48:15] elukey: Heya [13:48:36] o/ [13:48:37] 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4167967 (10Milimetric) [13:49:11] elukey: How has it been going on the reimage side? [13:50:07] elukey: guessing from https://grafana-admin.wikimedia.org/dashboard/db/analytics-hadoop?panelId=26&fullscreen&orgId=1&from=now-1h&to=now [13:50:11] joal: about 30m to be completed [13:50:30] elukey: if you want, I'm happy for druid later [13:51:03] joal: ack! Maybe in ~30m? [13:51:15] +1 elukey [13:58:36] joal: I am thinking a bit better about timings: if we don't care about Banner Impressions, we can stop it and then do the upgrade in a short timeframe, otherwise we'd need to wait one hour to allow each middlemanager to "Drain" from real time tasks [13:58:54] and there are a lot of meetings :) [14:01:06] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4167988 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka2002.codfw.wmnet'] ``` and were **ALL** succ... [14:02:42] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4167990 (10Ottomata) [14:02:57] PROBLEM - Hadoop DataNode on analytics1050 is CRITICAL: NRPE: Command check_hadoop-hdfs-datanode not defined [14:03:18] known, reimainging --^ [14:03:58] added some downtime [14:07:56] fdans: sorry, one more thing about https://gerrit.wikimedia.org/r/#/c/428390/7/modules/geoip/files/archive.sh why the "./" in cp -rl "$MAXMIND_DB_SOURCE_DIR/." ? [14:09:10] ottomata: hm, I thought it was necessary, but I guess not with -rl [14:09:17] 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4168012 (10Milimetric) I'm all for renaming to Geoeditors, but at this point it does involve a bunch of busy work, so I want to make sure everyone agrees, including Asaf. > Talking about over-nesting, there is also https://wik... [14:09:56] elukey: no prob for stopping job [14:10:33] fdans not sure what it would do? [14:10:43] the . [14:11:10] (if it were rsync) i would just end both args in / [14:11:12] since it doesnt' hurt [14:11:23] and generally means to copy the content, but will also create $CURRENT_DIR if it doesnt' exist [14:11:49] not sure if it has a different meaning without / with cp, but it does with rsync [14:11:50] so i just use it [14:11:59] i'd end both args with '/', but i dont' know what the [14:12:02] '.' would do [14:13:19] ottomata: I was copying all the contents of the directory with cp -R /path/ro/dir/. /path/to/newdir [14:13:26] but I think that's OSX's cp [14:13:45] do explicitly do content only, i'd seen '/*', but not '/.' [14:14:00] . afaik always refers to current dir [14:14:04] (could be wrong though) [14:14:35] ottomata: soooo i remove the dot? [14:15:02] fdans: [14:15:02] https://askubuntu.com/questions/86822/how-can-i-copy-the-contents-of-a-folder-to-another-folder-in-a-different-directo [14:15:07] i guess i just don't know about it! [14:15:40] let's leave it! [14:16:10] fdans: you've tested this script as is (with different args passed in on CLI) on stat1005 or somewhere? [14:17:56] 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4168038 (10Milimetric) > * It's worth documenting how distinct anonymous edits are counted. I assumed it was (user agent, IP) pairs, but it looks like it's just by IP. Done > * I suggest renaming whole dataset to something li... [14:18:14] joal: I've disabled notifications for the banner impression alert, if you want we can stop banner impression [14:18:23] I guess disable the cron on an1003 and kill the job? [14:18:24] elukey: moving forward now? [14:18:30] yeah let's do it [14:18:33] correct elukey [14:18:36] ok [14:18:48] elukey: I let you the cron, I do yarn [14:19:01] ack [14:19:11] 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban: Add pivot parameter to tabular layout graphs - https://phabricator.wikimedia.org/T126279#4168043 (10Milimetric) [14:19:14] elukey: I'm assuming it also needs a puppet manual steop, no? [14:19:20] To make sure cron stays disabled [14:20:15] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168050 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['ka... [14:20:37] joal: I disabled puppet on the host [14:20:42] k [14:21:19] !log Kill BannerImpressionStream job before upgrading druid [14:21:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:22:02] elukey: I also need to pause druid hourly loading jobs [14:22:05] elukey: doing so [14:22:57] super [14:23:33] !log disabled cron/check on analytics1003 to respawn banner impressions if needed [14:23:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:23:37] !log Suspend webrequest-druid-hourly-coord and pageview-druid-hourly-coord before druid upgrade [14:23:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:24:04] joal: in the meantime, two new workers again in service [14:24:33] gc timings for the namenode are awesome :) [14:24:42] elukey: checking metrics in NN while they get back in - looks super great indeed [14:25:10] RECOVERY - Hadoop DataNode on analytics1050 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [14:25:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4168060 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1049.eqiad.wmnet', 'analytics1050.eqiad.wmnet'] ``` and were **ALL** su... [14:29:59] joal: ready to upgrade druid1001 [14:30:05] I'll follow http://druid.io/docs/0.10.0/operations/rolling-updates.html [14:30:19] elukey: ack! [14:31:02] elukey: I'm assuming you'll do all historiocal nodes first, then all overlord etc? [14:32:31] joal: I was thinking one host at the time (all daemons on it), but I can follow your procedure as well [14:32:34] no preference [14:33:39] elukey: I have no clue if it changes anyhing [14:33:50] elukey: I'm assuming yours would be faster [14:34:58] joal: I am going to follow your way, I like it more [14:35:04] seems more consistent with the docs [14:35:08] ok :) [14:35:18] so druid1001's historical is alive [14:35:31] going to check its logs, when it finishes loading I'll do druid1002 [14:35:39] I wouldn't ever have guessed that one day my views would be "more consistent with the docs" :D [14:36:34] it wasn't an offence but a compliment :P [14:37:52] proceeding with druid1002 [14:39:11] Get:1 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main druid-common all 0.10.0-3~jessie1 [189 MB] :P [14:45:11] and upgrading 1003 [14:45:38] so 1002 went fine, 1001 complained a bit about segments not owned and other things [14:45:56] elukey: :( [14:46:12] elukey: nothing major I assume, anymore details? [14:46:51] ottomata: hadn't tested this last iteration, sorry, pushing now a tested in stat1005 modification [14:49:31] joal: the majority of them are [14:49:32] Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id 'hdfs' into a subtype of [simple type, class io.druid.segment.loading.LoadSpec] at [Source: N/A; line: -1, column: -1] [14:50:05] but now that I check, when I restarted the daemon it didn't log "loading segment bla bla" for all the things needed as druid100[23] [14:50:39] another interesting one is [14:50:39] 2018-04-30T14:40:04,660 ERROR io.druid.server.coordination.ZkCoordinator: Failed to load segment for dataSource: [14:50:42] that ends with [14:50:51] dataSource='webrequest', binaryVersion='9'}} [14:51:01] hm [14:51:21] bizarre! I'm looking at coordinator, and it tells me everything is fine :( [14:51:40] joal: segments loaded in druid1001? [14:52:11] I don't know about specific machines, but it tells me all segments for all datasources are loaded [14:52:19] And that we have 3 nodes [14:53:06] When looking at datasources, it shows segments from any workers (1,2,3) [14:53:19] elukey: looks good to me is what that means :) [14:54:01] yeah now logs seems a bit better [14:54:07] going to wait a bit before proceeding [14:56:19] elukey: still have historical for d1003, right? [14:56:39] already done [14:56:47] Wow - Fastman ! [14:56:54] historicals are on 0.10 now :) [14:57:14] elukey: and I can haz dataz - Looks good [14:57:57] mmmm still seeing those errors [14:58:02] I am wondering if https://gerrit.wikimedia.org/r/#/c/355469/ was applied [14:58:33] fdans: let's talk about El tests on standup, we should probably fix issues with jenkins [14:59:31] elukey@druid1001:~$ ls -l /usr/share/druid/extensions/druid-hdfs-storage-cdh/druid-hdfs-storage.jar [14:59:35] lrwxrwxrwx 1 root root 76 Apr 30 14:34 /usr/share/druid/extensions/druid-hdfs-storage-cdh/druid-hdfs-storage.jar -> /usr/share/druid/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.0.jar [14:59:39] seems fine [15:00:08] elukey: on namenode side - There still is a bump in old-gen GC after nodes come back alive, but man, they now take ~200ms :) [15:00:15] \o/ [15:00:57] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168150 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka2003.codfw.wmnet'] ``` and were **ALL** succ... [15:01:00] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168151 (10Ottomata) [15:01:06] ping mforns elukey [15:01:09] standduppp [15:02:48] joal: https://etherpad.wikimedia.org/p/analytics-druid-upgrade [15:04:12] is there anything pushing popups events to druid1001? [15:06:00] elukey: those segments are for 2017... [15:06:02] hm [15:08:21] joal: and only for druid1001 [15:08:27] weird ! [15:08:30] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168191 (10Ottomata) Done for main-codfw. Will proceed with main-eqiad this afternoon. [15:08:31] elukey: version difference? [15:09:02] (03CR) 10Mforns: [C: 032] Modify output defaults for EventLoggingSanitization.scala [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429427 (https://phabricator.wikimedia.org/T190202) (owner: 10Mforns) [15:14:49] (03Merged) 10jenkins-bot: Modify output defaults for EventLoggingSanitization.scala [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429427 (https://phabricator.wikimedia.org/T190202) (owner: 10Mforns) [15:20:53] joal: using a very scientific approach (a brutal restart of historical on druid1001) it loads segments now [15:20:58] (03CR) 10Joal: "Bump :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405053 (owner: 10Joal) [15:32:15] joal: I restarted the overlords (1001 is the new leader) and now I am doing middle managers [15:32:23] ack ! [15:36:15] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats bug: deploy causes bad merges - https://phabricator.wikimedia.org/T192890#4168280 (10Milimetric) 05Open>03declined We're just going to wait until we deploy with SCAP [15:37:33] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Hadoop HDFS Namenode shutdown on 26/04/2018 - https://phabricator.wikimedia.org/T193257#4168283 (10elukey) {F17441015} [15:40:01] joal: brokers done, last ones are coordinators [15:40:08] k [15:40:13] Those are the importan ones [15:42:16] 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4168294 (10Milimetric) I ran a quick check, and for 2018-03, there are 16% more distinct UA + IP combinations than distinct IPs, so that seems like a worthwhile change. cc @Nuria [15:42:30] (03CR) 10Nuria: "Let's talk with team how to best manage this datasource config (hiera? aqs alone?) before we merge this code." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/429765 (https://phabricator.wikimedia.org/T193387) (owner: 10Joal) [15:44:15] joal: how do you access to the coordinator ui? [15:44:33] tunnel to master coord on port 8081 [15:44:36] ssh -L 8081:localhost:8081 druid1002.eqiad.wmnet -N [15:44:38] perfect [15:44:38] elukey: --^ [15:44:55] elukey: up for me, looks good [15:44:56] I had the wrong one [15:45:03] druid1001 was not responding :P [15:46:28] ping milimetric [15:46:37] ping joal [15:47:42] joal: all upgraded \o/ [15:47:51] \o/ elukey ! [15:49:27] elukey: resuming hourly jobs? [15:50:24] joal: yep! [15:53:31] !log Resume webrequest-druid-hourly-coord and pageview-druid-hourly-coord [15:53:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:01:14] Success elukey :) [16:01:27] elukey: Do you mind restarting realtime before moving to ops meeting? [16:08:05] joal: yep doing it now! [16:08:13] thanks elukey :) [16:10:29] re-enabled the cron, it should create the job soon! [16:11:59] elukey: Checked the realtime job - it's failing - I'm gonna launch a manual version of it [16:13:21] :( [16:16:14] elukey: I can see 2 cron about BannerImpression on an1003!! [16:16:41] oooofff [16:18:15] joal: pretty sure it is a pebkac.. the process spawns every 5 mins or runs continously? [16:18:38] it theory every 5 min.. [16:18:38] mmm [16:19:34] joal: wait a min, I can see two things related to BannerImpression but one is the child of the other (bash and java) [16:19:37] no? [16:20:27] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 4 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168440 (10Tgr) [16:21:26] and from the overlord console I can't see failures.. [16:21:53] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 4 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168443 (10Pchelolo) > The problem started within hours of Kafka being enabled on mediawikiwiki, and it affects the wiki that's after mediawikiwiki a... [16:23:20] elukey: I think the first one failed because of the 2 crons [16:23:34] elukey: new job seems to work (I continue to monitor) [16:24:22] no idea why there were two [16:25:26] elukey: supposedly temporary patch :( [16:27:52] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168446 (10mobrovac) >>! In T193254#4168443, @Pchelolo wrote: > I believe this is the case. When switching the job to Kafka it was done only for test... [16:27:58] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4168449 (10mobrovac) [16:28:04] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168448 (10mobrovac) [16:30:47] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 10Services (watching): EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230#4168453 (10fdans) p:05High>03Triage [16:32:58] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 6 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168457 (10fdans) p:05High>03Triage [16:33:43] hey hey [16:33:49] can i get the context for these ^ ? [16:33:50] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Unify, if possible, AQS and Restbase's cassandra dashboards - https://phabricator.wikimedia.org/T193017#4157303 (10fdans) p:05Normal>03Low [16:34:06] why are you de-triaging tickets? [16:34:08] what is going on? [16:34:10] fdans: ^ ? [16:34:10] Hi mobrovac - We're in grooming, and were saying that we actually don't know [16:34:17] 10Analytics: reportupdater TLC - https://phabricator.wikimedia.org/T193167#4168469 (10mforns) p:05Triage>03Normal [16:34:21] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 6 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168466 (10Milimetric) p:05Triage>03High sorry - reverting accidental change of priority [16:34:51] joal: im sorry but you cannot do that for tickets like https://phabricator.wikimedia.org/T193254 [16:35:16] mobrovac: We didn't meant to deprioritize nor change, we have a bug in our process [16:35:32] mobrovac: issue is that eventbus tickets always tag analytics [16:35:47] mobrovac: it'll be corrected, we changed the priority of he ticket back [16:35:57] k thnx [16:36:13] np mobrovac - Sprry for the noise [16:36:23] while i'm here, why is https://phabricator.wikimedia.org/T193230 being put on "radar" [16:36:33] i was hoping you guys would look into that? [16:36:50] mobrovac: i think that was because i was not present in that tasking meeting so couldn't give context [16:36:54] i'll try to find some time to do that [16:37:04] joal: we should put that in ops excellence column [16:37:06] hehehehehe [16:37:08] Thanks for showing up ottomata :) [16:37:09] kk thnx ottomata! [16:41:19] elukey: could we clean that crontab? [16:43:59] joal: there was indeed a duplicated entry, maybe a copy/pasta error from me, not really sure (I checked carefully and ran puppet when I re-enabled) [16:44:02] sorry :( [16:44:16] elukey: I think it was Andrew, a long ago - nevermind :) [16:45:01] ran puppet, cron looks clean [16:45:05] The first one was actaully incorrect :) [16:45:09] Awesome - Tahnks [16:46:32] ahh nice I can see real time tasks! [16:46:43] * elukey dances [16:47:49] ;) [16:47:51] 10Analytics: [reportupdater] Add a configurable hive client - https://phabricator.wikimedia.org/T193169#4168542 (10mforns) p:05Triage>03Normal [16:48:11] elukey: I had to restart a manual job, for the same error we were having before we used the overlord-drain process [16:48:17] 10Analytics, 10Analytics-Wikistats, 10ORES, 10Scoring-platform-team: Discuss Wikistats integration for ORES - https://phabricator.wikimedia.org/T184479#4168545 (10Milimetric) ping @awight still want to chat about this? [16:48:20] elukey: everything back to normal :) [16:48:28] 10Analytics: [reportupdater] eliminate the funnel parameter - https://phabricator.wikimedia.org/T193170#4168547 (10mforns) p:05Triage>03Normal [16:49:40] 10Analytics: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171#4168554 (10mforns) p:05Triage>03Low [16:49:49] 10Analytics, 10Commons, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Make gwtoolsetUploadMediafileJob JSON-serializable - https://phabricator.wikimedia.org/T192946#4168557 (10Milimetric) p:05Triage>03Normal sorry - reverting accidental change of priority [16:50:23] 10Analytics: [reportupdater] consider not requiring date as a first colum of query/script results - https://phabricator.wikimedia.org/T193174#4168562 (10mforns) p:05Triage>03Normal [16:50:26] 10Analytics, 10Collaboration-Team-Triage, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Make EchoNotification job JSON-serialiable - https://phabricator.wikimedia.org/T192945#4168560 (10Milimetric) p:05Triage>03Normal sorry - reverting accidental change of priority [16:52:22] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 10Services (watching): EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230#4168578 (10mforns) p:05Triage>03Normal [16:52:55] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 10Services (watching): EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230#4168580 (10mforns) p:05Normal>03High [16:55:33] joal: tomorrow you are off right? [16:55:47] elukey: normally yes, I'll be on/off, but not working a lot [16:55:55] Maybe a bit more in the evening [16:56:06] elukey: I think we can call the druid upgrade a success :) [16:56:07] ah no I meant if it is public holiday in france [16:56:17] elukey: yes, public holiday inded [16:56:43] nice! so if you are ok I'd propose to upgrade druid100[456] on Wed [16:56:53] works for me elukey :) [16:56:57] thanks ! [16:56:58] super :) [16:57:11] after that I'll start working on 0.11 [16:57:20] elukey: <3 [16:57:23] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4168618 (10mobrovac) [16:57:38] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4168614 (10mobrovac) 05Open>03Resolved a:03mobrovac We have switched the LocalRenameUserJob for all wikis to EventBus, so we don't anticipate a... [16:57:42] Thanks again also elukey for the super productive change on NN today - This is great :) [16:58:01] joal: well it was a 50/50 effort, thank you too :) [17:02:07] 10Analytics: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4168647 (10Milimetric) [17:02:11] 10Analytics: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4168658 (10Milimetric) p:05Triage>03Normal [17:02:19] 10Analytics: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4168647 (10Milimetric) p:05Normal>03High [17:21:39] 10Analytics, 10Analytics-Wikistats, 10ORES, 10Scoring-platform-team: Discuss Wikistats integration for ORES - https://phabricator.wikimedia.org/T184479#4168727 (10awight) @Milimetric Sorry to miss the earlier ping. Some top-level metrics we can offer, (@Halfak please chime in here): * Total number of pred... [17:24:05] going afk! [17:24:07] * elukey off! [17:50:40] Hi! Quick question: how should I change the names of fields of existing schema? Or maybe I shouldn't do that? https://meta.wikimedia.org/w/index.php?title=Schema:MobileWikiAppDailyStats&diff=17983055&oldid=17836915 [17:53:46] chelsyx: unfortunetly you can't quite do it like that [17:53:52] because some of those fields are currently 'required' [17:53:54] so [17:54:02] we only support adding new fields [17:54:05] new optional fields [17:54:14] you can remove old optional fields, and nothing will break [17:54:33] but since those two fields are marked as 'required' [17:54:50] you can only add the new fields .e.g 'app_install_id', but you can't remove 'appInstallId' [17:56:32] Got you. Thank you! I revert that edit [18:19:59] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168953 (10Ottomata) [18:21:43] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4168964 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['ka... [18:23:39] (03PS1) 10Joal: Update changelog.md for v0.0.63 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429854 [18:24:15] fdans: while my kafka reimages are happening [18:24:29] shall we merge the geoip patch? [18:24:39] (03CR) 10Joal: [C: 032] Make stats gathering optional in mediawiki-history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427067 (owner: 10Joal) [18:32:42] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169027 (10alanajjar) @mobrovac I think it still the same! see [[https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Menageross03|here]], t... [18:33:07] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4169039 (10alanajjar) [18:33:11] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169038 (10alanajjar) 05Resolved>03Open [18:35:01] (03Merged) 10jenkins-bot: Make stats gathering optional in mediawiki-history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427067 (owner: 10Joal) [18:37:08] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4169045 (10alanajjar) [18:37:16] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169044 (10alanajjar) 05Open>03Resolved [18:37:54] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4164240 (10alanajjar) Thanks a lot all [18:38:51] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169059 (10Pchelolo) > As you know, we can't say it resolved until we being sure, because there's many pending requests, so if we said to all global... [18:39:19] (03PS2) 10Joal: Update changelog.md for v0.0.63 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429854 [18:39:47] ottomata: if you have a minue --^ [18:39:55] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169060 (10alanajjar) Yes @Pchelolo I noticed that now, Thanks again [18:40:10] (03CR) 10Ottomata: [C: 031] Update changelog.md for v0.0.63 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429854 (owner: 10Joal) [18:40:14] hanks ottomata [18:40:19] :) [18:40:34] (03CR) 10Joal: [V: 032 C: 032] "Merging before deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429854 (owner: 10Joal) [18:43:37] !log Releasing refinery-source [18:43:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:49:09] fdans: have you tested the dry-run feature in prod? [18:58:05] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4169111 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka1001.eqiad.wmnet'] ``` and were **ALL** succ... [18:59:03] ottomata: Heya - I have an issue with archiva - Jenkins tells me bad gateway when trying to upload [18:59:13] ottomata: Have you seen that before? [18:59:46] hm no [18:59:52] bad gateway? [18:59:54] lemme tail logs [18:59:58] https://integration.wikimedia.org/ci/job/analytics-refinery-release/103/org.wikimedia.analytics.refinery$refinery/console [19:00:01] ottomata: --^ [19:02:22] weird joal it did get uploaded though [19:02:29] hm [19:02:31] PUT /repository/releases/org/wikimedia/analytics/refinery/core/refinery-core/0.0.63/refinery-core-0.0.63.jar HTTP/1.0" 201 [19:02:37] ottomata: Shall I restart the job? [19:02:38] and i cna dl the file [19:02:39] i guess? [19:02:48] ok, trying again [19:10:54] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4169126 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['ka... [19:24:06] ottomata: I'm now facing a git issue, tag having been created :( [19:24:19] ottomata: Should I bump the tag, or is there a way to delete it? [19:25:18] i think we can delete [19:25:33] you might be able to joal [19:25:33] try [19:25:41] git push gerrit :v0.0.63 [19:25:43] will try ottomata [19:25:44] or whatever the tag is [19:27:14] 10Analytics: Rename new geowiki to geoeditors - https://phabricator.wikimedia.org/T193429#4169205 (10Nuria) [19:27:28] 10Analytics: Rename new geowiki to geoeditors - https://phabricator.wikimedia.org/T193429#4169215 (10Nuria) [19:27:32] 10Analytics-Kanban: Private geo wiki data in new analytics stack - https://phabricator.wikimedia.org/T176996#4169214 (10Nuria) [19:28:38] 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4169222 (10Nuria) OK ,agreed to change datasource name to new and shinny geoeditors. Please @Ijon let us know if you have something against it. Changed superset and wikis thus far [19:32:54] (03PS1) 10Joal: Reverting maven commits after failed deploy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429861 [19:32:58] ottomata: it worked :) [19:33:00] Thanks ! [19:33:22] (03CR) 10Joal: [V: 032 C: 032] "Merging before deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429861 (owner: 10Joal) [19:34:40] !log Retry releasing refinery-source to archiva [19:34:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:52:39] ottomata: failed for a different reason ... I'm gonna clean the things, and then try again on wednesday [19:53:30] ok [19:53:32] sorry joal :/ [19:53:40] np ottomata - Thanks for the help [19:54:32] (03PS1) 10Joal: Reverting again maven commits after failed deploy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429867 [19:54:53] (03CR) 10Joal: [V: 032 C: 032] "Merging for clean state" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/429867 (owner: 10Joal) [19:55:42] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4169394 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka1002.eqiad.wmnet'] ``` and were **ALL** succ... [19:59:17] Gone for tonight after 2 failed deploy :( [20:02:49] 10Analytics: Weird performance of sqoop job on Edit Reconstruction - https://phabricator.wikimedia.org/T172579#4169423 (10Milimetric) 05Resolved>03Open no, this issue is unrelated to the changes we've made, and likely to still be a problem. [20:05:10] joal: can i help you with deployment? [20:43:22] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (AKA Event(Logging) of the Future (EoF)) - https://phabricator.wikimedia.org/T185233#4169592 (10Pchelolo) [21:21:38] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169694 (10MarcoAurelio) Do we need to migrate `CentralAuthRename` too? If so, can it be done? Thanks. [21:24:07] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169698 (10Pchelolo) > Do we need to migrate CentralAuthRename too? If so, can it be done? Thanks. Eventually everything will be migrated. Are you s... [21:27:06] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169701 (10Tgr) That's a log channel, not a job queue. Other potentially affected jobs are LocalUserMergeJob (not sure if Wikimedia wikis still allow... [21:28:13] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169704 (10Tgr) >>! In T193254#4169701, @Tgr wrote: > LocalPageMoveJob (I think that's triggered differently, not quite sure though). Yes it is. So... [21:31:58] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169715 (10MarcoAurelio) We are not performing any user account merges nor globally nor locally. Regards. [21:36:15] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169720 (10Tgr) Other instances of cross-wiki job scheduling that are yielded by a quick `ack 'JobQueueGroup::singleton\( '`: Cognate/LocalJobSubmitJ... [21:46:02] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169741 (10Pchelolo) > Other instances of cross-wiki job scheduling that are yielded by a quick ack 'JobQueueGroup::singleton\( ': Cognate/LocalJobSu... [22:06:29] 10Analytics, 10EventBus, 10GlobalRename, 10MediaWiki-JobQueue, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254#4169799 (10MarcoAurelio) Is this related to T192604 anyhow? Regards. [22:29:51] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade to Stretch and Java 8 for Kafka main cluster - https://phabricator.wikimedia.org/T192832#4169832 (10Ottomata) only kafka1003 remains...will do tomorrow. [22:36:47] 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4169845 (10Tbayer) >>! In T191343#4168012, @Milimetric wrote: > I'm all for renaming to Geoeditors, but at this point it does involve a bunch of busy work, so I want to make sure everyone agrees, including Asaf. > >> Talking a... [22:38:36] Ah, good, should do that for all Archive pages, note to self [22:45:54] PROBLEM - Kafka MirrorMaker main-eqiad_to_eqiad average message consume rate in last 30m on einsteinium is CRITICAL: 0 le 0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_eqiad [22:52:08] milimetric: i just remember that i did that on our pages on medoiawiki.org after Nemo_bis told me, had totally forgot [22:52:13] *forgotten [22:52:55] oh yeah, I vaguely remember that, so I forgot even more :) [22:55:07] !log bouncing kafka main-eqiad -> eqiad (analytics) mirror maker [22:55:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:56:11] ottomata: doesn't seem to be an alert, but around the same time it looks like https://stream.wikimedia.org/v2/stream/recentchange started 502'ing [22:56:30] seems plausibly related, although i'm unsure the architecture exactly [22:57:41] 502ing?! [22:57:45] that is likely related [22:57:50] it lives on this mirror maker instance [22:57:55] RECOVERY - Kafka MirrorMaker main-eqiad_to_eqiad average message consume rate in last 30m on einsteinium is OK: (C)0 le (W)100 le 7892 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_eqiad [22:57:55] i might take job queue stuff out of this one too [22:58:36] looks to have come back now, lines up well :) [22:59:07] yeah, but it isn't going to stay [22:59:15] ebernhardson: how did you notice the eventstream 502? [22:59:34] ottomata: i have a silly thing in labs that watches the stream and generates image signatures for a a sampling of uploads [22:59:49] it got the 502's [22:59:57] s/labs/cloud/ [23:00:12] ah hm ok [23:01:27] (03PS2) 10Nuria: [WIP] UA parser specification changes for OS version [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/429527 (https://phabricator.wikimedia.org/T189230) [23:03:30] PROBLEM - Kafka MirrorMaker main-eqiad_to_eqiad max lag in last 10 minutes on einsteinium is CRITICAL: 6.603e+05 gt 1e+05 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_eqiad [23:04:34] !log blacklisting change-prop and job queue topics from main-eqiad -> analytics (eqiad) [23:04:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:05:02] although, i don't seem to be receiving any changes still. it only stopped 502 [23:05:11] yeah [23:05:16] i dunno why that would 502 on this problem [23:05:22] but messages are stuck ^^^ should help [23:05:31] give it a few [23:16:53] ebernhardson: should be ok now [23:17:31] RECOVERY - Kafka MirrorMaker main-eqiad_to_eqiad max lag in last 10 minutes on einsteinium is OK: (C)1e+05 gt (W)1e+04 gt 0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_eqiad [23:17:48] that is ok! [23:17:49] i will ack that [23:17:58] it is going to report that for a week, until the now blacklisted topics expire [23:18:01] OH [23:18:02] recovery! [23:18:51] oh, the change-prop & job topcis are alreayd blacklisted from alerting [23:18:52] ok cool [23:19:01] so yeah, lag on e.g. recentchange stream fixed