[01:42:43] 10DBA, 10Wikimedia-Site-requests: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3350501 (10MZMcBride) >>! In T48098#3020665, @Dereckson wrote: > @jcrespo Do you have some thoughts on this topic? I'd like to create a list of ten s3 wikis, one... [05:23:59] 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3350572 (10Marostegui) [05:29:41] 10DBA, 10Labs: Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3350575 (10Marostegui) Hi, I have sanitized this wiki on sanitarium and sanitarium2 and checked that it sanitized labs servers. #cloud-services-team can you guys go ahead and create the views? [05:36:16] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3350578 (10Marostegui) @demon if you think `searchindex` can be safely truncated, could you create a subtask for it here? Thank you! [05:40:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3350580 (10Marostegui) [05:43:51] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3350584 (10Marostegui) #cloud-services-team the alter table on labsdb1003 - s4 instance finished and it is now catching up with the master. [06:08:43] 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3350593 (10Marostegui) #cloud-services-team please note that this alter has been started on labsdb1003 so delays are expected on the `s2` instance for a couple of days or so. [06:12:40] 10DBA, 10Labs, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3350594 (10Marostegui) #cloud-services-team feel free to create the views for dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 Thanks! [07:41:17] 10DBA, 10Wikimedia-Site-requests: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3350636 (10jcrespo) please note s3 <> small, there are some very large wikis on s3, like cebwiki, which are scheduled to be given more resources, but it happens... [08:19:40] 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3350679 (10Marostegui) [09:17:17] 10DBA, 10Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3350827 (10Marostegui) @Ottomata @elukey what do you guys want to do with this? [09:30:38] 10DBA, 10Labs, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3350858 (10JAllemandou) Yay ! New wikis ! Thanks @Marostegui and teams :) [10:54:54] 10DBA, 10RESTBase, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 2 others: Investigate requirements for MySQL access in RESTBase - https://phabricator.wikimedia.org/T164805#3351022 (10Tgr) [11:34:27] I am leaving pc1004 importing/replication: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=pc1004&from=now-1h&to=now [11:36:11] https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=pc1004&var-network=bond0&from=now-24h&to=now [13:45:19] 10DBA, 10Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3351373 (10Ottomata) @elukey might have other opinions, but I'm inclined to try our best to expedite the ordering of new hardware, rather than worry about the BBU. I... [14:16:45] 10DBA, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351418 (10jcrespo) [14:23:51] 10DBA, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351450 (10Marostegui) Should the missing user be created then? [14:24:58] 10DBA, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351455 (10jcrespo) Is there a file where grants for this are tracked? Should it be shared between labswiki and labstestwiki? [14:43:53] 10DBA, 10Operations, 10cloud-services-team, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351529 (10Marostegui) The only file where I could see it was on `wikitech.sql.erb` and if we add it to labstest I would suggest we add it there... [14:44:33] 10DBA, 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3351533 (10Halfak) a:03Halfak [14:48:43] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2070 - https://phabricator.wikimedia.org/T167667#3351589 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete. [14:50:36] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2070 - https://phabricator.wikimedia.org/T167667#3351593 (10Marostegui) Thanks! Will close the ticket once the rebuilt is done: ``` logicaldrive 1 (3.3 TB, RAID 1+0, Recovering, 2% complete) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600... [14:51:19] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351613 (10jcrespo) [14:54:26] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351637 (10jcrespo) maybe we should use the core grants file? or does wikitech have other different grants from core? if ye... [14:56:22] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351639 (10Marostegui) >>! In T167961#3351637, @jcrespo wrote: > maybe we should use the core grants file? or does wikitech... [15:01:59] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351653 (10jcrespo) We do not have to wait, is it easy to see if everthing on core covers wikitech? What about the rest- I... [15:08:00] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351662 (10Marostegui) >>! In T167961#3351653, @jcrespo wrote: > We do not have to wait, is it easy to see if everthing on... [15:13:37] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351678 (10jcrespo) Something oauth from horizon seem to be the largest blocker, what do you know about that? How would that fit with the main... [15:16:50] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351686 (10jcrespo) [15:18:01] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351232 (10jcrespo) [15:19:46] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351594 (10bd808) >>! In T167973#3351678, @jcrespo wrote: > Something oauth from horizon seem to be the largest blocker, what do you know about... [15:19:52] 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review, 10Wikimedia-log-errors: Cronjobs attempting to connect to labstestweb2001 - https://phabricator.wikimedia.org/T167961#3351714 (10jcrespo) [15:23:24] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351594 (10Andrew) I am pretty sure that this is fine. I would like to be present and alert during the switchover, though, in case I'm forgett... [15:24:56] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351725 (10jcrespo) That- which is a really good thing for separation of concerns (Thank you!)- would mean that the generic grants would work (... [15:31:04] sorry sorry for confusing you [15:31:21] I didn't want an opinion from me to shut you down, marostegui [15:31:30] haha no worries! [15:31:51] I said "I am not sure" [15:32:02] but I didn't -1 on purpose because I really wasn't sure [15:32:25] it turned out the best of the options, probably [15:32:32] :) [15:33:44] apologies again [16:00:08] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3351824 (10demon) [16:00:11] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3351836 (10demon) [16:00:15] marostegui: As requested ^ [16:00:24] Thank you! [16:03:07] profiling, job, updatelog [16:03:11] All could use subtasks [16:03:43] interwiki [16:21:25] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Idh0854 → Garam: supervision needed - https://phabricator.wikimedia.org/T167031#3351989 (10MarcoAurelio) 05Open>03stalled p:05Normal>03Low Unfortunately busy days are ahead and I'll not be able to handle this request. My plan is to deci... [17:27:38] 10DBA, 10Operations, 10Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3352203 (10jcrespo) [17:28:07] 10DBA, 10Operations, 10Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3156297 (10jcrespo) taking also db1099 and db1101 T167567 [22:19:08] 10DBA, 10Labs, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3353325 (10bd808) [22:30:25] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353371 (10bd808) @Marostegui can you do the magic needed to fix the "Access denied for user 'maintainviews'@'localhost' to database 'dewik... [22:32:17] 10DBA, 10Labs: Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3341927 (10bd808) @Marostegui need some help still: `pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'atjwiki_p'")`. That's from labsdb1009 but I imagine t... [22:32:51] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3353377 (10bd808) a:03bd808 [22:35:00] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353312 (10jcrespo) I have run on labsdb1009: ``` GRANT SELECT, SHOW VIEW ON `dewiki\_p`.* to labsdbuser; GRANT SELECT, SHOW VIEW ON `wikid... [22:36:29] jynus: labsdb1009 is still not happy. the script is trying to do CREATE DATABASE ... [22:36:47] that's as the maintainviews user [22:37:44] 10DBA, 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#2669300 (10Halfak) Somehow I accidentally added yuvi. :S So I've attempted to load the table once and I had... [22:44:58] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353391 (10jcrespo) So we have a problem with the script-- the script has enough privileges to create the database, but only after the data... [22:49:50] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353398 (10bd808) >>! In T168021#3353391, @jcrespo wrote: > Not sure how to fix the loop. It looks like the script expects to create the D... [22:51:25] but that makes no sense- our users create databases with no problem [22:54:52] theer is also no global create: https://dev.mysql.com/doc/refman/5.7/en/grant.html#grant-global-privileges [22:55:18] I think there is a bug on grants using wildcards [22:56:56] and we cannot create grants for non-existing databases [22:57:15] and that user is already a sysadmin (super) [22:59:06] jynus: hmmm... I jsut ran it again on labsdb1009 and it seems to have worked, but it also said the views were all there [22:59:20] yeah, I run one test [22:59:31] but the point was for you to be able to do it on your own [22:59:37] right [22:59:46] and the script is limited on purpose [23:00:21] not because of you, but because we want limited accounts everywhere to avoid security issues in such a public infrastructure [23:00:30] understood [23:00:52] * bd808 does not feel persecuted by jynus :) [23:01:00] so adding more grants defeats the hole purpose [23:01:46] I can run the script this time and I will think how to fix this [23:02:08] so right now the issue is that you are granting CREATE for the new db name but things don't work until someone else has actually created it? [23:02:46] I think yes, that grants do not kick in if they are in the shape of '%string' or '%string%' [23:03:02] because this method works for user created dbs [23:03:21] ah. an user grants would work possibly because they are uid%_p [23:03:22] things s54353_mydb on old labsdbs [23:03:31] it is my thesis [23:03:34] or uid_% or whatever [23:03:38] but only that would explain it [23:03:55] I am looking at documentation, and I think it is a bug [23:04:23] let me try one thing [23:04:49] I am going to add a grant for wikidatawiki% [23:04:54] and we will test there [23:04:58] *nod* [23:05:32] try now wikidatawiki on 1009 [23:05:41] that worked [23:05:52] ok, I am going to drop it [23:06:47] try again, the same [23:06:58] "Access denied for user 'maintainviews'@'localhost' to database 'wikidatawiki_p'" [23:07:08] yeah, 99% bug confirmed [23:07:18] from the CREATE DATABASE call [23:07:19] *sure [23:07:37] let me give it a thought [23:07:53] and I will figure out how to solve it [23:07:57] gross work around... make a%, b%, c%, ... z% grants [23:08:04] ugh [23:08:12] I said gross :) [23:08:27] it is one way [23:08:52] just give me some time, I will think on it, report it upstream, etc. [23:09:44] yup. should we open a phab bug about the general problem? [23:09:56] or is there one already? [23:10:04] I just figure it out [23:10:15] just copy 4 lines to the ticket you were working [23:10:23] and I will think about it tomorrow [23:10:52] I am happy because the bug was suspected [23:11:08] but never knew exactly why it happened [23:12:32] I can think of a way to workaround it [23:13:13] I can create a stored procedure that executes as root [23:13:45] it is still a workaround, but much cleaner [23:14:14] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353464 (10bd808) a:05bd808>03jcrespo @jcrespo and I debugged this a bit on irc and found that a grant for `wikidatawiki%` worked to al... [23:15:54] A stored proc would be nicer than the 26 grants for sure. I always forget that mysql grew up and got things like that years and years ago [23:16:07] well [23:16:21] I am not going to dissapoint you [23:16:40] but there is a reason why this is python and not the horrible mysql scripting syntax [23:16:42] :-) [23:16:47] :) [23:17:39] I spent 7+ years writing PL/SQL. I have a hardened sense of what is reasonable for a db scripting [23:18:10] the whole application db layer at $DAYJOB-1 was stored procs (grosss) [23:18:29] pl/sql is a programming language [23:18:48] mysql is... text that happens to be turing-complete? [23:19:03] well I also edit mediawiki wikis :) [23:20:00] yes, parserlua modules is comparable, indeed [23:22:17] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353482 (10jcrespo) I used `%wik%\_p` but `%\_p` doesn't work either. [23:24:03] and it is on the specific case of creation [23:24:24] because we also do %\_p for users [23:24:36] but for select