[06:25:13] 10DBA, 10cloud-services-team (Kanban): Drop nova and nova_api databases from m5 - https://phabricator.wikimedia.org/T248313 (10Marostegui) Thank you - I have renamed this on the rest of the slaves: db1117:3325, db2135, db2078:3325 On Monday I will drop them [08:29:54] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [09:21:47] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [09:50:51] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1130 [] db1124 [] db1113 [] db1110 [] db1102 [] db1100 [] db1097 [] db1096 [] db1082 [10:13:03] Hugo! [10:13:07] Hiyo ... [10:14:38] Would it be possible to for example change the name of the current wb_terms on labs to something else, but then also put a new empty wb_terms table on labs? With the same old name? [10:14:43] Or would that break something? [10:15:48] why not creating a view? [10:16:12] With the old structure but new tables data? [10:16:20] Could investigate it! [10:16:47] yep [10:18:49] We did plane on investigating a view at some point, but not sure what happened with it as it happened while I was away [10:18:52] addshore: the view as jaime says is an option of course, placing a new empty wb_terms can also be done [10:19:04] They are different approaches and you'd get different results of course :) [10:19:25] I might investigate a view quickly [10:20:42] addshore: can I quote you publicly without saying your name "I've talked to wd people and they are going to investigate it" [10:24:47] ^addshore [10:24:59] In a call about it currently ;) [10:25:04] ok [10:33:54] maybe dont advertise that we are looking yet, Lea of WMDE is preping another announcement now [10:34:02] so, let me look at creating this view though... [10:34:45] so, basically I need to write a single sql query that would give me a table, that looks like the old one? [10:35:14] yep, look at that file with other examples [10:35:32] "that file with other examples" ? [10:35:39] I am looking for it [10:35:59] thanks! [10:38:33] addshore: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/labs/db/views/maintain-views.yaml [10:39:32] e.g.: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/labs/db/views/maintain-views.yaml$660 [10:45:36] im guessing a union would be evil? ;) [10:49:29] well, not evely, but as far as I know, the split was done now with a join, so I see no point [10:49:32] *evil [10:49:53] if there was vertical split, sure, why not [10:56:07] something like this mayber [10:56:10] https://www.irccloud.com/pastebin/9nNhyncs/ [10:56:39] * addshore scratches head to see if the joins re correct .... [10:56:58] Is that returning all rows from wbt_item_terms and wbt_property_terms is a big question :P [10:57:02] run that and run the same with different LIMIT offset, number; [10:57:09] or order by [10:57:19] on the real table and you can compare [10:57:43] offset 1000000 is sloooowww [10:57:46] yeah [10:58:00] try order by [10:58:18] or just deploy, and we can tune later? [10:58:42] xD [10:59:00] well, my thought is, this view might be totally useless for actual queries [10:59:09] so there might be no point at all in adding it [10:59:25] just like wb_terms!!! [10:59:38] that is why it should have an expiration date [10:59:42] to prevent bad queries [10:59:52] but the data would still be there [11:00:07] check a query by id [11:00:14] and that should be "fast" on both [11:00:56] if I use that select and then try to filter down to selecting the en label for P31, it just.... doesnt return :P [11:02:30] I can see why, because of the concat [11:04:43] :D [11:05:01] without the concat it might be better then? [11:05:20] no, the concat is needed [11:49:31] does some magic caching happen once the query is actually made to be a view? [11:49:50] Tried out a query on the anlytics db server and it took (24 min 7.57 sec) to return 1 label for one property :D [12:34:19] Hi again! so the view wont work :) [12:34:25] is there such a thing as a table alias [12:34:33] wait, i think I just answered my own question...... [12:35:22] Would we make a view of wb_terms exactly as it currently is on labs called wb_terms_no_longer_updated ? [12:35:40] then on monday we would remove the wb_terms_no_longer_updated view, move rename wb_terms to wb_terms_no_longer_updated and then create an empty wb_terms in the old place? [12:35:48] jynus: ^^? :) [12:36:17] then what do you want the view for? [12:36:40] for just the time between now and monday? [12:36:46] so that people can change their code now to point to wb_terms_no_longer_updated [12:36:57] otherwise they have to do it at the moment that we change the table name [12:37:02] ah, I see [12:37:03] now they can do it anytime between now and monday [12:37:04] :) [12:37:08] if that would work? :D [12:37:15] Up to you really :) [12:37:29] Renaming and creating a new table on Monday is easy [12:37:52] I think though, for the views creation, better to coordinate with the Cloud Team [12:38:05] They do maintain the views and all that, so I think they should be aware of all that [12:38:18] okay! [12:38:24] I suggest creating a task with high priority and ping bstorm_ on that task [12:38:37] Please include me too, so I can coordinate the table renaming and creation [12:39:14] ack! [12:49:46] messaged them in a secret IRC channel first [12:49:53] then would make a phab ticket if they say yes [12:57:33] cool [12:57:34] thanks [14:30:34] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) a:03jcrespo Running connect manually: ` 'success': False, 'database': None, 'errno': 1043, 'port': 3306, 'errmsg': 'Bad handshake' ` [14:36:49] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) https://bugs.mysql.com/bug.php?id=91828 https://github.com/aio-libs/aiomysql/issues/85 Protocol change or connector version? Will try on a buster host. [14:47:06] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) ` root@db1114:~$ python3 Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from WMFMariaDB import WMFMaria... [14:54:45] I just realise that DROP INDEX IF EXISTS is only a feature mariadb has and not mysql? [14:55:14] yeah, they aded a buch of IF EXISTS in the past [14:55:21] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) 05Open→03Resolved We need a version of the mysql connector higher than python3-pymysql 0.9.8 for compatibility of the new mysql authentication protocol on latest versions. Either backporting, or... [14:55:25] 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [14:55:37] ^I have resolved this, as per ticket scope [14:56:19] Oh nice! [14:56:21] Thanks [14:56:25] just to be clear [14:56:29] I have resolved the ticket [14:56:31] not the issue [14:56:53] yeah, I read the summary [14:57:02] ok, because the resolution was not as nice :-D [14:57:11] "just wait" [15:00:50] https://youtu.be/E8_ARd4oKiI?t=143 [15:01:47] haha [15:02:03] I was expecting a new release from your gifs [15:02:06] But youtube is ok [15:02:13] But please note that it has been ages since you released a new one [15:02:29] https://youtu.be/E8_ARd4oKiI?t=143 [15:20:34] 3:18 PM addshore: writes don't affect the views. It's reads that makes this take a long time. It's like 20 min of fussing in an ideal case on a view that isn't used by anybody often. The time goes up very quickly if I end up having to manually depool replica servers and then kill a lot of connections to get the lock to rebuild the views across all databases...like it can be hours out of the day [15:20:37] jynus: ^^ thoughts ? :D [15:22:24] addshore: the issue is, this will take either a little bit or significant time on my part...why am I doing it? :) [15:22:34] I don't know why I am not just doing it on Monday [15:23:17] why not creating it manually? [15:23:26] it is 1 table, right? [15:23:33] 1 view [15:24:07] or you mean dropping the old one? [15:24:23] Wait...I just got working and haven't had any caffeine: why change the old one at all? Why don't I just add a new view with the new name??? [15:24:47] whatever works, we had some of those with the _compat name [15:24:56] she will know the details [15:26:29] So, the idea is people can start using the table with the new name over the weekend and riday, rather than on monday doing the rename and everyone else needing to change their code on multiple tools etc at the same time [15:27:03] just to be clear, we can help with anything you both need- I just think I am an unuseful extra step here- cloud will know the operational part of wikireplicas better than us :-D [15:27:23] 0:-) [15:27:25] addshore: so why not add the new view with the new name instead of changing the old one? It'll be the same amount of work for me either way because the lock is on the underlying table. [15:27:43] "Why don't I just add a new view with the new name", because on monday we will empty what is in the wb_terms place, so that tools stop reading the old data, and then keep it in "wb_terms_no_longer_updated" for a period if tools dont mind the data being out of date, but then that is a decision made by the tool owner [15:28:02] or, desperate measures call for desperate times- kill all connections using it live :-D [15:28:13] I do however generally agree this seems like an unuseful extra step. The place where I get ahead of things in views is when the schema change breaks code. [15:28:19] hmm, wait, isnt everything already a view in the labs replicas or something? :P [15:28:33] that's true [15:28:35] jynus: been doing that a lot lately...there's always something dangling if the revision table or a couple other busy tables is involved [15:28:46] addshore: yeah, users only access views [15:28:53] Right!!!! :P [15:28:58] see- you do know the env better than we do :-D [15:29:01] yes so actually we just need a new view with a new name today [15:29:16] then on monday we just remove the old wb_terms view and put a new one there that is empty [15:30:41] I'll be straight with you: I'm not thrilled with a high priority shift in my work for a minor convenience vs fixing something broken. I'm just checking why we need to do it today when we also need to do it Monday. :) [15:30:50] that's why I suggested to get WMCS in the loop, they play with the views more than we do :) [15:30:59] But if it really helps a lot of people, I'll do it! [15:32:30] addshore: I'd totally make the ticket...easier to explain there as well :) I should be able to get you set up. I just wanted you to know that it can take me a fair bit of time in some instances so can we make sure it's really worth it to you and other users first. [15:33:02] I'll make a ticket now! [15:33:25] how many replicas in cloud land does wikidatawiki exist on? [15:33:27] Ok thanks! :) [15:33:46] just wikidatawiki? That's on the three servers [15:33:56] yup, just wikidatawiki :) [15:33:59] If it's just wikidatawiki, it's a lot easier. I didn't have that context LOL [15:34:15] :) [15:34:16] yup, sorrry, *adds that context now* :D [15:35:07] let's make it on labsdb1009-labsdb1012 (for consistency) I guess? [15:35:30] does this need to go into testwikidatawiki or no need, addshore ? [15:35:33] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10MoritzMuehlenhoff) The Cumin update is something that will happen next Q, but if you need it earlier, we can backport it as well? [15:35:37] we can ignore testwikidatawiki [15:35:41] ok [15:38:46] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) Not worth the work. This is not blocking anything important- that is why I just closed it. This was a FYI, to both cumin and/or deb monitor owners. This was the first time I realized this incompatibi... [15:39:55] created https://phabricator.wikimedia.org/T248592 [15:39:55] 10DBA, 10Cloud-Services, 10Wikidata: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) [15:44:36] bstorm_: I dont want to be pushy, but any idea of an ETA of when we might be able to make that view occour today? :) [15:47:36] In a quick meeting... [15:47:40] brb [15:48:15] ack! :) [15:50:38] Something is broken and am helping with the fix [15:58:27] Ok! back. I think I can get to it later this morning. Since it's just one wiki, I think it will be fairly quick. 😁 [15:58:59] "later my morning" is more correct...I'm in basically PST [16:00:08] marostegui: thanks for the reminder on 1012. I would have remembered when I started, but I said "three servers" as a knee-jerk :) [16:12:34] #AnalyticsWikireplicaInstanceMatters [16:28:41] amazing!! [16:43:00] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) [16:43:48] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) a:03Bstorm [16:51:46] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) @Addshore the last step on Monday I guess you meant create an empty `table`? Also when do you want the table rename to happen? [16:55:00] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10WMDE-leszek) [16:55:43] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10WMDE-leszek) FWIW, I've updated the description to the date of the second step to match the one announce to the community. [17:14:13] 10DBA, 10Patch-For-Review: wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError - https://phabricator.wikimedia.org/T218189 (10jcrespo) Sorry this is taking so long- many people decided to send their GSoC proposal at the last possible minute, so there is only so much time I can expen... [17:33:29] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) >>! In T248592#6002548, @Marostegui wrote: > @Addshore the last step on Monday I guess you meant create an empty... [17:46:46] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) So as far as I understand and from https://gerrit.wikimedia.org/r/583693 The view `wb_terms_no_longer_updated`... [17:49:14] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) I would prefer the table rename be done, then the views can be adjusted so that wb_terms and wb_terms_no_longer_up... [17:51:51] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) Generally, anything that is "select *" is under the fullviews structure in our config as long as the table name i... [17:52:50] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) Proposing a change to the description to match that.... [17:53:56] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) p:05Triage→03Medium [17:54:16] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) Look good now @Addshore and @Marostegui ? [17:54:42] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) [17:58:12] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) That sounds good to me Thank you :) [18:09:31] 10DBA, 10Wikidata, 10Patch-For-Review, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) labsdb1011 is fighting with me. Lots of wikidata queries going on there. I'll start killin' 🔪 [18:16:56] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) Done with today's part. Openstack upgrades seem to have affected some bots, or I'd put it in SAL. [18:17:12] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) [18:17:25] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) a:05Bstorm→03None [18:18:30] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Marostegui) @Addshore maybe this is not the right task to ask for it but...is Analytics aware of this change? They use labsdb1012 to sqoop data... [20:28:35] 10DBA, 10Patch-For-Review: wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError - https://phabricator.wikimedia.org/T218189 (10Guozr.im) Thank you for your time @jcrespo :) Also, feel lucky to finish the initial drafts of my proposal at an early time, so I could work on some of the s... [20:36:53] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Addshore) >>! In T248592#6002784, @Marostegui wrote: > So as far as I understand and from https://gerrit.wikimedia.org/r/583693 > The view `wb_t... [22:05:02] 10DBA, 10Wikidata, 10cloud-services-team (Kanban): Move wb_terms data in cloud replicas to wb_terms_no_longer_updated - https://phabricator.wikimedia.org/T248592 (10Bstorm) >>! In T248592#6003342, @Addshore wrote: > > wb_terms in the cloud replicas for wikidatawiki_p is a view though? Or did I get that wron...