[05:08:05] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3405812 (10Marostegui) This happened again ``` root@db1046:~# megacli -LDInfo -Lall -aALL | grep Policy Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write... [05:16:41] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3405816 (10Marostegui) And it recovered for now: ``` ˜/icinga-wm 7:15> RECOVERY - MegaRAID on db1046 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy root@d... [07:08:12] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3405971 (10Marostegui) It alerted again, but this time looks like the BBU is actually doing the learning: ``` BatteryType: BBU Voltage: 3754 mV Current: -674 mA Tempe... [09:08:54] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3406297 (10Marostegui) `˜/icinga-wm 11:05> RECOVERY - MegaRAID on db1046 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy` [09:52:40] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Monitor that no worker nodes are in the default rack in net topology - https://phabricator.wikimedia.org/T163909#3406408 (10elukey) [09:53:00] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Monitor that no worker nodes are in the default rack in net topology - https://phabricator.wikimedia.org/T163909#3214047 (10elukey) a:03elukey [10:39:02] * elukey lunch! [11:52:40] Hi a-team [11:53:01] hello joal how is it going???? [11:53:01] I have a confirmation now: I'll take my month off from now on ! [11:53:27] joal: confirmation? :D [11:53:41] Our daughter Naé is born this morning; July 5th, at 5:24 :) [11:54:08] Mother and child are doing good but need rest :) [11:54:08] awwwwww congratulations joseph [11:54:12] this is so wonderful [11:54:20] It isvery much ! [11:54:26] Thanks fdans :) [11:55:04] * joal is feeling like like a free-hugs provider for at least a few days [11:55:50] joal: congratulationsssssssss!!!! \o/ [11:56:47] Hi elukey :) [11:56:48] Thanks [11:58:31] joal: how is lino doing after all these emotions?? [11:59:04] elukey: So far so good, meeting with Naé for the first time this evening :) [11:59:11] :) [11:59:51] elukey: Mentioning to him he was brother of a little sister and showing him a picture generated very positive reactions, so I'm happy already :) [12:03:03] congratulations! [12:15:40] Thanks moritzm :) [12:16:46] a-team: I cleaned a bit the kanban from my tasks [12:17:04] The ones in CR are still there, one in dne is actually really closed (uniques) [12:17:27] Two from done have been moved to Paused, + a new one created about sqoop jobs failing [12:19:13] anything to handover? [12:19:37] 10Analytics-Kanban: Sqoop job fqil for big wiki tables - https://phabricator.wikimedia.org/T169735#3407032 (10JAllemandou) [12:20:00] elukey: this one (we discussed it yesterday): https://phabricator.wikimedia.org/T169735 [12:20:14] Changing the parameter in JDBC url didn' work [12:20:20] More investigation needed. [12:20:55] elukey: the one on hadoop queue, you know it (but no emergency, so however you prefer) [12:22:01] elukey: The normalized_host.project_family needs to be merged and deploy (deploy guide in task, tricky one) [12:22:34] And lastly the one on splitting webrequest tags is I think reasonably easy to grab from any of you guys :) [12:22:37] how urgent is the sqoop one? [12:23:12] since you are here, can you update the task description with repro steps? [12:23:29] (also with a note about what credentials to use) [12:23:36] (for labsdb-analytics) [12:24:13] the hadoop queue might be delayed until you are back, not really urgent and it will need vigilant eyes after the deployment :) [12:26:14] completely agreed on hadoop queue [12:27:39] elukey: I have a repro command for sqoop at home, will update the task later tonight [12:30:09] joal: ah sorry I thought you were at home now! [12:30:16] sure sure whenever you have time :) [12:30:43] elukey: no bother :) [12:31:12] elukey: please send me a reminder later this week if I forget to update (possible given the context) [12:31:50] elukey: Sqoop one is q blocker for the new fields and timestamps in mediawiki_history, but more importantly for new snapshot finalisation [12:32:04] I'll try to be precise enough in my update on the task [12:33:46] elukey, mforns: how is it going with EL purge? A bit better orshould I just shut up? [12:36:37] Leaving for now a-team, will be here and there once every now and then :) [12:38:26] o/ [12:38:34] EL purging is.... slowly proceeding :P [12:40:24] joal: godspeed! so excited for you :D [12:54:34] hey yall [12:56:09] joal: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [12:56:19] congratulations man, don't worry about cleaning stuff up, we got it [12:56:47] halfak: we meeting? Joseph's not gonna be there, and I have other stuff I could do, but happy to meet if you've got something burning [13:01:40] o/ Nothing on the agenda for me [13:03:07] milimetric: o/ [13:03:09] confirmed with others on he call. Skipping todays meeting! [13:03:20] thanks halfak [13:03:31] fdans: gimme a few minutes to catch up and then can I catch up with you? [13:03:41] ssure [13:13:16] fdans: ok, batcave? [13:33:25] helloooooo [13:33:40] elukey, white-list is ready to go [13:34:34] mforns: o/ - I am finishing the alters on db1047, and checking if the remaining tables have all fields whitelisted [13:34:53] elukey, ok! need help? [13:35:18] should not but I'll let you know otherwise! [13:35:36] k [13:59:02] oh fdans do we know what we need to do in webpack to get the computed: { ...mapGetters(['something']) } syntax work? [13:59:10] the ... part seems to not compile with the current setup [13:59:51] damn [14:03:03] milimetric: https://github.com/babel/babel-loader/issues/170 [14:03:34] fdans: oh good: "It starts working when i add transform-object-rest-spread to plugins" [14:11:13] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3407478 (10MarcoAurelio) Per above. [14:34:37] 10Analytics-Tech-community-metrics: Merge detached Phab and MediaWiki accounts with the same username in DB - https://phabricator.wikimedia.org/T169754#3407552 (10Aklapper) [14:37:03] fdans: I just pushed - the tests were broken, fixed now [14:37:13] awesome, pulling [14:37:17] fdans: and you have an error in DimensionalData testing [14:37:22] in breakdown [14:37:31] and I didn't wanna fix because I didn't know how that works in the Detail [15:05:41] 10Analytics, 10ChangeProp, 10EventBus, 10Epic, and 2 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3407608 (10mobrovac) [15:31:42] ping elukey [16:19:54] msg milimetric let's have 1 on 1 in 5 mins? [16:21:24] nuria_: yeah, sure [16:34:44] * elukey off! (dentist, sigh) [17:24:37] oh I forgot - I gotta go to a routine checkup with Steph in a half hour [17:24:41] I'll be working later tonight [17:42:57] I also have to pick someone at the airport in 15, will be working later as well [17:52:06] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 4 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3408409 (10Nuria) [17:53:19] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 5 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3324826 (10Nuria) Then, if we want to drop tables ( I agree, that is the best solution) this is the path... [17:56:57] milimetric: you did see: https://gerrit.wikimedia.org/r/#/c/363136/ [17:57:24] milimetric: and : https://gerrit.wikimedia.org/r/#/c/363163/ a [17:58:15] 10Analytics-Kanban: Troubleshoot issues with scoop of data not working - https://phabricator.wikimedia.org/T169782#3408433 (10Nuria) [17:58:32] 10Analytics-Kanban: Troubleshoot issues with scoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3408448 (10Nuria) [17:58:43] 10Analytics-Kanban: Troubleshoot issues with scoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3408433 (10Nuria) a:05JAllemandou>03Milimetric [17:58:55] 10Analytics-Kanban: Troubleshoot issues with scoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3408433 (10Nuria) Please see: https://gerrit.wikimedia.org/r/#/c/363163/ [17:59:17] 10Analytics-Kanban: Troubleshoot issues with scoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3408433 (10Nuria) [17:59:29] milimetric: created ticket and assigned it to you: https://phabricator.wikimedia.org/T169782 [18:02:03] thanks nuria, just read the patches and will get back to troubleshooting [18:02:20] gotta go to the doctor's now, I'll be back at 6 my time [18:02:31] milimetric: k! [18:52:03] 10Analytics: Define what constitutes a search pageview - https://phabricator.wikimedia.org/T120249#3408672 (10Nuria) 05Open>03declined [19:11:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 6 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3408803 (10Nuria) [19:31:26] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819#3408912 (10Smalyshev) [21:05:28] k, back [21:05:32] looking at sqoop thing [21:06:07] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3409364 (10Milimetric) [21:32:13] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3409468 (10Milimetric) Diagnosis so far: It looks like Joseph did something close to the following sequence of actions: * add two tables to the s... [21:33:00] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for data lake data with addition of two new tables - https://phabricator.wikimedia.org/T169782#3409474 (10Milimetric) [21:33:02] 10Analytics-Kanban: Sqoop job fqil for big wiki tables - https://phabricator.wikimedia.org/T169735#3407032 (10Milimetric) [21:33:40] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3408433 (10Milimetric) [21:52:24] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3409533 (10Milimetric) This is where the log of the error shows up in hue btw: https://hue.wikimedia.org/jobbrowser/jobs/job_1498042433999_48036/job_attempt_logs/0 [22:01:43] 10Analytics-Kanban: Sqoop job fqil for big wiki tables - https://phabricator.wikimedia.org/T169735#3409566 (10Milimetric) 05duplicate>03Resolved Joseph - if you do end up updating this, no worries on too much detail. I get how the error is happening and will see how to increase that net_write_timeout or fin... [22:06:12] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3409588 (10Milimetric) more context: it looks like Bob West ran into the same issue a couple of years ago while reading from analytics-store: https://wm-bot.wmflabs.org/logs/%23wikime... [22:33:23] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 4 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3409666 (10kaldari) @Ottomata: Who is capable of actually doing this? [22:36:50] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 4 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3409674 (10Nuria) @kaldari: either @elukey or @Ottomata can do it but we just need to ping DBa... [22:43:59] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3409695 (10kaldari)