[02:30:14] <wikibugs>	 Analytics-Kanban, EventBus, Wikimedia-Stream: Public Event Streams - https://phabricator.wikimedia.org/T130651#2658074 (Ottomata) Hm, true.  But you know, it would be pretty trivial to have both options at different URLs.  I just hacked this together in service-template-node...it works.  Haven't gott...
[06:09:05] <wikibugs>	 Analytics-Kanban: Varnishkafka should auto-reconnect to abandoned VSM - https://phabricator.wikimedia.org/T138747#2658204 (elukey) p:Low>Normal
[06:41:15] <elukey>	 joal: o/
[06:41:28] <elukey>	 if you are ok I'd stop camus and oozie bundles
[06:41:35] <elukey>	 as prep step for the reboots
[06:45:04] <elukey>	 !log stopped camus on analytics1027 and suspended webrequest-load-bundle via Hue (prep step for reboots)
[06:45:08] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[06:45:16] <elukey>	 let me know if it is enough :)
[06:46:05] * elukey commutes to the office
[07:11:17] * elukey back
[07:12:23] <elukey>	 mmm the spark shell app master issue in Yarn UI seems to be related to a 200 with CL 0
[07:15:32] <elukey>	 but then it goes to 503
[07:15:42] <elukey>	 and I don't see anything like that in the apache logs
[07:42:05] <elukey>	 now I can still see stuff on stat1002/3 running in cron and screen for different people
[07:44:36] <elukey>	 neilpquinn, halfak - I can see screen sessions and cron jobs with your username on stat1003
[07:44:52] <elukey>	 (probably you are not awake but it is worth to try)
[07:46:51] <elukey>	 and there is an apache running on stat1002?
[07:48:31] <elukey>	 same thing on 1003
[07:52:59] <elukey>	 sorry to everybody, just rebooted stat100[23] :/
[07:54:07] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Allow AKlapper to access https://wikimedia.biterg.io/edit/ - https://phabricator.wikimedia.org/T144704#2658301 (Lcanasdiaz) a:Dicortazar>Lcanasdiaz
[08:11:33] <elukey>	 Starting the reboot of the Hadoop cluster
[08:52:24] <elukey>	 !log varnishkafka 1.0.12 installed in cache:upload esams
[08:52:27] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[08:56:54] <joal>	 Hi elukey
[08:57:10] <joal>	 No alerts from what I see on Hadoop, sounds good
[08:58:29] <elukey>	 hello :)
[08:58:37] <elukey>	 I am restarting 104* atm
[08:59:03] <elukey>	 I can see some jobs killed in  yarn but could it be due to the stat restarts?
[08:59:18] <elukey>	 or should those have kept running independently?
[08:59:33] <elukey>	 (user jobs, mostly milimetric and mforns)
[09:14:06] <elukey>	 !log varnishkafka 1.0.12 installed in cache:upload codfw
[09:14:08] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[09:32:35] <elukey>	 joal: I think that we'd need to reboot 1027 and 1003 as well
[09:32:49] <elukey>	 so stopping oozie and hive* daemons
[09:39:09] <elukey>	 !log rebooted analytics1027
[09:39:12] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[09:43:39] <elukey>	 ah nice hue does not come up
[09:43:41] <elukey>	 grrr
[09:47:52] <elukey>	 I forget all the times about the apache instance that comes up and steals port 8888 to hue
[09:47:55] <elukey>	 grrr
[09:47:59] <elukey>	 need to nuke it
[09:49:24] <elukey>	 !log suspended all oozie bundles as prep step to reboot analytics1003
[09:49:26] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[09:51:54] <elukey>	 !log executed aptitude remove apache2 on analytic1027 (we use nginx in front of hue, apache steals port 8888 to hue and it does not start)
[09:51:57] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[09:58:50] <elukey>	 so user 'west1' in hadoop is creating tables with hive
[09:58:58] <elukey>	 is he in this chat by any chance?
[09:59:05] <elukey>	 because I'd need to restart hive and oozie
[10:04:07] <elukey>	 !log rebooted analytics1003 (oozie, hive-metastore and hive-server2 daemons affected)
[10:04:10] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[10:04:24] <elukey>	 last ones are 100[12]
[10:11:38] <elukey>	 ah snap the mysql database
[10:11:51] <elukey>	 IIRC the last time we asked to Jaime to fix it
[10:12:55] <elukey>	 The MariaDB server is running with the --read-only option so it cannot execute this statement
[10:14:54] <joal>	 elukey: I think there is a SQL statement that unlock it (like SET something = true;)
[10:15:14] <elukey>	 yeah but I don't have it written anywhere
[10:15:23] <joal>	 neither do I :(
[10:18:10] <elukey>	 I stopped oozie as we did last time
[10:18:13] <elukey>	 maybe hive too
[10:21:20] <icinga-wm>	 PROBLEM - Oozie Server on analytics1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.catalina.startup.Bootstrap
[10:23:03] <elukey>	 yeah I know oozie, but it is better that you stay down
[10:23:27] <elukey>	 Jaime is in a meeting, I'll ping him later on
[10:25:27] <elukey>	 so SET GLOBAL read_only = 0; should do the trick but I am not sure if we'd need to do other commands or not
[10:25:31] <elukey>	 very ignorant
[10:25:55] <joal>	 elukey: IIRC it's the only thing
[10:28:27] <elukey>	 we can try
[10:28:50] <elukey>	 !log setting global read_only = 0 to analytics1003 mariadb instance
[10:28:52] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[10:29:35] <elukey>	 all good :)
[10:32:00] <elukey>	 now 1001 and 1002 are the only ones left to reboot
[10:32:36] <icinga-wm>	 RECOVERY - Oozie Server on analytics1003 is OK: PROCS OK: 1 process with command name java, args org.apache.catalina.startup.Bootstrap
[10:32:36] <elukey>	 joal: would you mind to rapidly check if all is good?
[10:33:52] <joal>	 elukey: eqi1004
[10:33:55] <joal>	 ops :)
[10:35:41] <joal>	 elukey: Can't connect to hive from stat1004
[10:36:16] <joal>	 elukey: hue looks ok from oozie perspective
[10:36:29] <elukey>	 this one sounds familiar
[10:36:39] <elukey>	 is it a deja vu from the past reboots?
[10:36:49] <joal>	 I can't recall elukey
[10:36:53] <joal>	 hm
[10:37:09] <elukey>	 so hive uses the metastore right?
[10:37:10] <joal>	 elukey: have you restarted camus?
[10:37:11] <elukey>	 on 1003
[10:37:18] <joal>	 correct elukey
[10:37:19] <elukey>	 joal: nope, it is stopped
[10:37:23] <joal>	 k
[10:37:36] <joal>	 (trying to check only what is needed :)
[10:38:50] <elukey>	 joal: now is it better?
[10:39:17] <joal>	 yes elukey, solved :)
[10:39:22] <joal>	 elukey: What was it?
[10:39:51] <elukey>	 I gently restarted the hive metastore :D
[10:39:59] <joal>	 :d
[10:40:01] <elukey>	 I think it was upset for some reason
[10:40:02] <joal>	 :D
[10:40:04] <joal>	 Great
[10:40:16] <joal>	 elukey: Possibly the mysql lock thing
[10:40:17] <elukey>	 NOOOWWWW
[10:40:23] <elukey>	 100[12] ?
[10:40:36] <joal>	 no prob for me, I don't have anything there
[10:40:40] <elukey>	 reboot 1002, wait, check, failover, reboot, etc..
[10:40:47] <joal>	 elukey: hadoop nodes have been restarted or not yet?
[10:41:50] <joal>	 elukey: If yes, I'd rather restart camus and oozie
[10:43:02] <elukey>	 yes all restarted
[10:43:21] <elukey>	 so I just stopped hdfs and yarn masters on 1002, and checked that 1001 is still the boss
[10:43:28] <elukey>	 rebooting 1002 should be a no-op
[10:44:00] <joal>	 Ahhhh elukey, analytics100[12], not stat100[12] !!!
[10:44:03] <joal>	 I get it :)
[10:44:13] <joal>	 sorry
[10:44:51] <elukey>	 okok! :D Maybe I wasn't clear earlier on sorry
[10:45:17] <joal>	 Nah, just me not making links :)
[10:45:23] <elukey>	 :)
[10:45:32] <elukey>	 so 1002 is up and running, yarn/hdfs are bootstrapping
[10:45:46] <joal>	 k
[10:46:57] <elukey>	 good, both daemons ready
[10:50:13] <elukey>	 so now I'll wait 15 minutes JUST TO BE SURE
[10:50:18] <elukey>	 and I'll failover 1001
[10:51:03] <joal>	 elukey: if ok, I'd rather wait 5 minutes, hadoop is gently delayed in the mean time
[10:51:13] <elukey>	 sure
[10:51:15] <joal>	 :-P
[10:51:18] <joal>	 Thanks ;)
[10:51:28] <elukey>	 I tried to do everything as fast as possible today :(
[10:51:48] <joal>	 elukey: That's absolutely ok, I'm sorry if I came pressuring
[10:52:01] <elukey>	 nono I got your point :)
[10:52:07] <joal>	 elukey: I'll shut up next time ;)
[10:52:14] <joal>	 15 minutes is not big
[10:52:29] <elukey>	 nono 5 are enough, ETOOPARANOIA
[10:52:34] <joal>	 :D
[10:52:55] <elukey>	 in the meantime the new vk is deployed in upload esams and codfw
[10:53:02] <elukey>	 seems working fine
[10:53:10] <joal>	 awesome, you rock :)
[10:53:11] <elukey>	 I'll deploy to ulsfo and eqiad this afternoon
[10:53:29] <elukey>	 Hope that this change will help
[10:55:09] <elukey>	 !log Failover from analytics1001 to analytics1002 as prep step for 1001's reboot
[10:55:11] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[10:57:32] <elukey>	 !log rebooted analytics1001
[10:57:35] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[11:00:44] <elukey>	 joal: everything done
[11:00:52] <joal>	 Great !
[11:01:07] <elukey>	 I am wondering if we should keep 1002 as master for a couple of hours, and then failover again this evening
[11:01:13] <elukey>	 just to avoid messing up too much
[11:01:20] <elukey>	 what do you think?
[11:01:40] <joal>	 I have no real opinion
[11:02:04] <joal>	 elukey: only 'issue-ish' thing is yarn.w.o broken ;)
[11:02:06] <joal>	 as expected
[11:02:34] <elukey>	 ahhhhh snap!
[11:02:40] <elukey>	 I forgot about that
[11:02:49] <joal>	 elukey: no big deal :)
[11:02:58] <joal>	 I can monitor CLI
[11:03:22] <elukey>	 ok let's do in this way, we can tunnel to 1002 for a bit
[11:03:30] <joal>	 sounds good elukey
[11:03:59] <elukey>	 all right re-enabling everything
[11:04:03] <elukey>	 oozie and camus
[11:04:10] <joal>	 k
[11:04:58] <elukey>	 !log re-enabling oozie and camus after cluster reboots
[11:05:00] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[11:10:19] <elukey>	 oozie started to complain about SLA miss, it is back :)
[11:10:43] <elukey>	 so follow ups: remove the read only at boot
[11:13:56] <joal>	 elukey: We should autmatically do that yeah
[11:18:23] <elukey>	 I am going afk for a bit to have lunch, please let me know on hangouts if anyhing goes on fire
[11:35:18] <mforns>	 hi team!
[11:35:26] <joal>	 Hey mforns
[11:36:21] <joal>	 mforns: currently testing, but I might have found a way through restores :)
[11:36:35] <mforns>	 joal, oh!
[11:36:36] <mforns>	 cool
[11:37:22] <mforns>	 joal, do you want to explain? :]
[11:37:29] <joal>	 I can mforns !
[11:37:38] <mforns>	 batcave?
[11:37:43] <joal>	 sure, OMW !
[12:28:41] <joal>	 mforns: I have a difficult case :(
[12:28:54] <mforns>	 joal, wanna cave?
[12:29:00] <joal>	 yup
[12:29:03] <mforns>	 omw
[12:47:55] <wikibugs>	 Analytics-Kanban, EventBus, Wikimedia-Stream: Public Event Streams - https://phabricator.wikimedia.org/T130651#2658705 (mobrovac) I'd be +1 for using SSE here alongside WebSockets. The URI structure could be straightforward for this - `/{topic}{/partition}{/offset}` in its simplest form. To add filte...
[12:48:39] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2658707 (mobrovac)
[12:50:52] <elukey>	 !log varnishkafka 1.0.12 installed in cache:upload ulsfo and eqiad
[12:50:54] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[12:51:01] <elukey>	 we are missing misc, maps and text
[12:51:23] <elukey>	 I'll try to do maps/misc today if I have time
[12:51:26] <elukey>	 and text tomorrow
[12:51:32] <elukey>	 but upload is covered
[13:12:50] <elukey>	 !log restarted oozie jobs for 2016-9-22-6
[13:12:52] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[13:16:07] <elukey>	 !log set read_only = false (on startup) for the analytics1003's mariadb instance
[13:16:09] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[13:16:51] <elukey>	 !log previous comment was meant to be read as "set a permanent read only = false"
[13:16:53] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[13:47:51] <wikibugs>	 Analytics-Dashiki, Analytics-Kanban, Need-volunteer: Vital-signs layout is broken - https://phabricator.wikimedia.org/T118846#2658817 (Milimetric) Todos for Dashiki, collected while doing this refactor  [] refactor code and style for out-of-service component [] organize the components folder into log...
[13:48:30] <milimetric>	 (I'm around, working on dashiki semantic upgrade)
[13:48:49] <elukey>	 (hello semantic Dan :)
[13:50:07] <milimetric>	 hi Luca, I thought you were taking today off?
[13:58:47] <elukey>	 nope, I'll take a couple of hours tomorrow to catch my flight
[13:59:24] <elukey>	 Did I tell you guys a different thing? :O
[13:59:53] <halfak>	 elukey, just saw your ping.  What happened with stat1003?
[14:00:06] <moritzm>	 halfak: it was rebooted for a kernel security update
[14:00:29] <halfak>	 Damn.  This was unusually bad timing.  I lost a lot of work.
[14:00:45] <elukey>	 :(
[14:01:19] <elukey>	 really sorry halfak, I noticed the screen session and the crons but I wasn't sure about them
[14:01:37] <elukey>	 yesterday I sent an email to analytics, research and engineering about the reboots
[14:01:58] <halfak>	 elukey, gotcha.  I should have seen and protested then.  No worries.
[14:02:08] <milimetric>	 halfak: is there a better way to notify?  Sorry you lost work
[14:02:18] <elukey>	 yeah I was about to ask the same
[14:02:49] <halfak>	 Not sure.  Let me look at the email.
[14:03:08] <milimetric>	 maybe like [REBOOTING] in the subject line?  and people can filter for it?
[14:03:34] <milimetric>	 elukey: related-ish question: the replication of data to eventlogging seems to have stopped yesterday
[14:03:34] <halfak>	 I don't see an email to research
[14:03:58] <halfak>	 Maybe it didn't go to the right email address.
[14:04:24] <joal>	 ottomata: Hi !
[14:04:33] <joal>	 ottomata: Would you have a minute for me?
[14:04:42] <milimetric>	 halfak: it's [Analytics] Upcoming reboots of stat and Hadoop hosts due to Kernel upgrades
[14:04:57] <milimetric>	 it went to wiki-research-l (forwarded by me) and analytics-l, then to engineering separately
[14:05:05] <halfak>	 milimetric, ahh.  It went to wiki-research-l!
[14:05:19] <milimetric>	 should we hit the internal one instead?
[14:05:24] <halfak>	 That might be the confusion.  Yeah.  I think so.
[14:05:39] <halfak>	 Still I should have seen it on the analytics lists.
[14:05:50] <halfak>	 Maybe we can have a direct ping list for those machines.
[14:06:17] <halfak>	 It's sort of like, E.g., you wanted to reboot ORES VMs and contacted me via labs-l
[14:06:41] <milimetric>	 yeah, it's hard without a smaller specific list
[14:06:43] <halfak>	 I think there's only a few of us who regularly use stat100(2|3) for long term jobs.
[14:07:04] <halfak>	 Maybe I can make one and suggest people put their contact on wikitech
[14:07:24] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2658860 (Ottomata) Actually, by both I didn't mean websockets and SSE.  I meant HTTP with SSE at one endpoint, and then HTTP on a non SSE endpoint, on which...
[14:07:33] <ottomata>	 joal:  hiya sure!
[14:07:44] <ottomata>	 oh we gotta do some puppet hting!
[14:07:51] <joal>	 ottomata: batcave?
[14:07:51] <milimetric>	 so I see two options: 1. make a new public list for notices like this or 2. craft subject lines such that people can set up filters and send a note about the convention
[14:07:56] <joal>	 or IRC?
[14:08:05] <ottomata>	 sho
[14:08:09] <ottomata>	 bc
[14:08:31] <milimetric>	 elukey: what I was going to ask is, do you know if that replication script runs on one of those stat boxes and needs to be restarted manually?
[14:08:40] <milimetric>	 (I'll ask in -databases too)
[14:09:10] <halfak>	 anyway, milimetric & elukey, thanks for the chat.  I'll think some more and come back with a proposal.
[14:09:28] <halfak>	 FWIW, I see missing this as my fault and something I should work on fixing.
[14:10:55] <elukey>	 I'll be available to chat about your proposal when you'll be ready :)
[14:11:24] <elukey>	 milimetric: mmmmm not that I know
[14:11:25] <elukey>	 checking
[14:11:40] <elukey>	 when did it stop?
[14:11:59] <milimetric>	 it looks like some tables are about 6-7 hours behind and one of them was 24 hours behind
[14:14:23] <elukey>	 ah milimetric it seems to be running for role::mariadb::analytics::custom_repl_slave, that IIRC is one of the analytics nodes
[14:14:26] <elukey>	 checking
[14:14:58] <elukey>	 no I was wrong :D
[14:17:17] <milimetric>	 come to -databases, elukey, jaime's looking at it
[14:18:52] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2658903 (mobrovac) To me, the most compelling reason for using WS are browsers. I share your feelings:  > But, I'm a backend engineer, and I betcha there are...
[14:42:51] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2659104 (Nuria) > I'm a backend engineer, and I betcha there are lots of folks who want to consume a stream in a browser. Answer is yes. Even if it is for "s...
[14:59:35] <joal>	 elukey: Do you mind pouting back an1001 as yarn master?
[14:59:51] <joal>	 it causes troubles accessing UIs (even with tunnels)
[15:01:02] <nuria_>	 mforns: milimetric standduppp
[15:06:00] <elukey>	 joal: sure :)
[15:08:52] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2659235 (Ottomata) > Why the latter? If we exclude for a moment browser-based solutions from the discussion, what would be the benefits in using the latter s...
[15:27:18] <wikibugs>	 Analytics: Kill limn1 - https://phabricator.wikimedia.org/T146308#2659277 (Nuria)
[15:30:44] <elukey>	 !log analytics1001 is back Yarn/HDFS master
[15:30:47] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[15:31:27] <elukey>	 joal: done :)
[15:32:33] <joal>	 thanks elukey :)
[15:41:14] <ottomata>	 joal:  [@analytics1027:/mnt/hdfs/user/hive] 2 $ ls
[15:41:15] <ottomata>	 hive-site.xml  warehouse
[15:41:47] <joal>	 Thanks a milion ottomata !!!
[15:42:04] <joal>	 Will test, then sublit CRs for refinery :)
[15:42:07] <joal>	 ottomata: --^
[15:42:30] <ottomata>	 k cool
[16:00:25] <ottomata>	 gotta run out to apt.  bbl
[16:16:21] <wikibugs>	 Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2659403 (Nuria)
[16:18:24] <wikibugs>	 Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2659406 (Nuria)
[17:13:31] <wikibugs>	 Analytics-Kanban: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2659496 (elukey)
[17:13:51] <wikibugs>	 Analytics, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2659497 (leila) @nuria re your comment on https://gerrit.wikimedia.org/r/#/c/311964/ about the IP address: it's...
[17:15:41] <wikibugs>	 Analytics-Kanban: Upload the final version of the pivot repo and load test - https://phabricator.wikimedia.org/T146389#2659499 (elukey)
[17:16:30] <wikibugs>	 Analytics-Kanban: Upload the final version of the pivot repo and load test - https://phabricator.wikimedia.org/T146389#2659499 (Nuria) Siege would be of help here: https://wikitech.wikimedia.org/wiki/Analytics/Siege
[17:17:05] <wikibugs>	 Analytics-Kanban: Upload the final version of the pivot repo and load test - https://phabricator.wikimedia.org/T146389#2659521 (Milimetric)
[17:26:35] <milimetric>	 joal: you wanted to brain bounce now?  isn't it late?
[17:27:01] <joal>	 I have 1/2h before my wife comes and shout at me ;)
[17:27:02] <joal>	 let's go for it !
[17:27:34] <joal>	 milimetric: --^
[17:27:40] <joal>	 milimetric: to the batcave !
[17:27:59] <milimetric>	 omw
[17:28:30] <joal>	 mforns: talking with milimetric in the cave about weird pages
[17:28:38] <joal>	 mforns: just if you fee like it :)
[17:28:52] <mforns>	 joal, there's this meeting in 2 minutes though
[17:29:01] <joal>	 oh yeah !
[17:29:03] <joal>	 no problemo
[17:31:36] <milimetric>	 nuria_: ping meeting
[17:31:41] <milimetric>	 nvm :)
[18:01:03] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: decom titanium - https://phabricator.wikimedia.org/T145666#2659725 (Dzahn) alright, handing it over to ops-eqiad now.   the server has been shutdown, public IP removed from DNS, removed from puppet/icinga etc.  please go ahead with physical decom steps, disk...
[18:01:19] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: decom titanium - https://phabricator.wikimedia.org/T145666#2659739 (Dzahn) a:Dzahn>None
[18:02:42] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: decom titanium - https://phabricator.wikimedia.org/T145666#2637714 (Dzahn) location: eqiad row B, B4 @ 13
[18:09:14] <ottomata>	 milimetric:  do you have context about future fields and events yall might need for mediawiki history project?
[18:09:23] <ottomata>	 there is a patch from collab team that is adding some fields
[18:09:30] <ottomata>	 want to make sure they make sense from yall's perspective
[18:16:55] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/312274/1
[20:15:34] <nuria_>	 ottomata if you are there.. could you please install siege in aqs1001?
[20:15:52] <nuria_>	 ottomata: i want to run load tests on the old hosts to sanity check numbers
[20:16:05] <ottomata>	 can do!
[20:16:10] <nuria_>	 ottomata: maythnaks sir
[20:16:13] <nuria_>	 *thanks
[20:16:30] <ottomata>	 done.
[20:18:34] <nuria_>	 wow
[20:18:36] <nuria_>	 k
[21:14:30] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2660509 (Afandian) I see a lot of browser-based discussion here. I'd just like to contribute my voice as an existing consumer of the existing RC Stream and h...
[21:17:19] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2660511 (Ottomata) @Afandian thanks for chiming in.  Could you comment on what is easier for you in Java / Python?  socket.io vs a more simple streamed HTTP...
[21:42:06] <wikibugs>	 Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1195035 (Quiddity) This feature would be incredibly helpful, IIUC. I have two tasks that require checking things across all our projects, and I don't know how else to do it, oth...
[21:47:28] <wikibugs>	 Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2660674 (GWicke) On the Java SSE client front, I see these two options:  - https://github.com/michaelklishin/eventsource-netty5 - https://jersey.java.net/api...
[22:39:16] <grrrit-wm>	 (PS1) Milimetric: Upgrade to semantic 2 everywhere [analytics/dashiki] - https://gerrit.wikimedia.org/r/312430 (https://phabricator.wikimedia.org/T118846)
[22:39:28] <grrrit-wm>	 (PS2) Milimetric: [WIP] Upgrade to semantic 2 everywhere [analytics/dashiki] - https://gerrit.wikimedia.org/r/312430 (https://phabricator.wikimedia.org/T118846)
[23:25:24] <grrrit-wm>	 (CR) Nuria: "Let me know when you think you are ready and I will test layout in chrome/ff/safari" [analytics/dashiki] - https://gerrit.wikimedia.org/r/312430 (https://phabricator.wikimedia.org/T118846) (owner: Milimetric)
[23:27:18] <grrrit-wm>	 (CR) Nuria: [WIP] Upgrade to semantic 2 everywhere (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/312430 (https://phabricator.wikimedia.org/T118846) (owner: Milimetric)
[23:43:28] <wikibugs>	 Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1195035 (yuvipanda) @Quiddity having some form of official resources dedicated to it might be helpful. I unfortunately don't think I'll have any bandwidth to be able to look at...