[06:17:04] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Marostegui) 05Open→03Stalled L'et stall it so we don't forget about it, and we can test the query plan with the following mariadb releases [08:23:04] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [08:25:17] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) After analyzing the indexes on T217397 and testing a few hosts (T217397#4997997 and T217397#5010633... [08:25:55] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) After analyzing the indexes on T217397 and testing a few hosts (T217397#4997997 and T217397#5010633) with the exact table definition on tables.sql, we are going to unify `loggin... [08:34:42] pc1010:9104 showing 1 collection failure [08:34:58] I rebooted it for upgrade [08:35:02] ah! [08:35:04] ok [08:35:12] (it is a spare also) [08:47:52] As there is no sre etherpad, I have added my stuff to our etherpad [08:48:37] ok, I can do that too [08:48:46] :) [08:48:52] * marostegui misses the stickers here XD [08:51:46] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) The following hosts on s1 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] dbstore1001... [08:51:48] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) The following hosts on s1 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labs... [08:54:27] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) The following hosts on s4 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1004 [] db1125 [] d... [08:54:30] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) The following hosts on s4 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labs... [08:56:09] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) The following hosts on s5 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1124 [] d... [08:56:13] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) The following hosts on s5 eqiad have the indexes unified with tables.sql (T217397#5010633) [] labs... [09:00:22] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [09:01:12] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [09:11:12] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [09:11:14] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [09:13:37] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [09:13:42] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [13:31:58] hi folks [13:32:19] our openstack deployments in the eqiad DC use a database in prod (m5-master) [13:32:43] any chance we can do the same in the codfw DC? [13:32:59] currently the openstack DB is running in a mysql instance in control servers [13:33:20] but we would like to have the deployments in codfw to be as close as possible to eqiad as possible [13:33:36] possible possible possible [13:34:31] we are planning on rebuilding all of the codfw stuff for openstack, that's why I'm mentioning it now [13:34:48] (https://phabricator.wikimedia.org/T217891 for context) [13:37:57] could you clarify what codfw deployment is? is it a replica of eqiad data or it is a separate thing? [13:38:07] is a separate thing [13:38:47] the codfw deployments are use to develop/experiment with openstack before we introduce features/bug fixes into the production Cloud VPS service. So, these codfw deployments aren't customer facing [13:39:33] so sure but it will need a new name [13:39:46] e.g. m6 or something [13:40:18] and either new hard or puppet reorg witha new instance on existing hardware [13:40:42] I am guessing if it is for testing it won't be very intensive? [13:40:59] resource-* [13:41:06] I don't expect so, but we had problems in the past bc the openstack setting was hammering the DB [13:41:10] what is the level of ha needed? [13:41:18] no HA needed [13:41:29] file a ticket, there is a set of questions we need you (service) to answer [13:42:08] ok, anyway I still have a conversation pending in my own team, bc I'm not sure we all agree in this topic [13:42:18] but assuming it is not resource intensive it should fit on existing hardware. configuration wise may be not immediate as we don't have any rw instance there [13:42:31] we also don't have proxies [13:42:42] I see [13:42:51] we we lack some machines and configuration, so don't expect this to be fast [13:43:25] ok [13:43:30] misc is not fully setup on codfw, so you know [13:43:46] there is data replication, but not active dbs or basic service at the moment [13:44:08] *misc active dbs [13:44:26] so you know that would impact ETA [13:45:29] we want to work on that, but if you need it *now* a separate db on eqiad would be suggested instead [13:45:41] no, I don't need it now [13:46:02] let me pass you the meta ticket so your team is aware of blockers, pending work [13:46:20] arturo: https://phabricator.wikimedia.org/T156937 [13:46:48] speciall, we hadn't work much with cloud services, because last time we asked: "Not ready or intended to switchover" [13:47:05] so it is even more delayed than gerrit or phab [13:47:31] well, technically this is not for switchover, but you get the idea [13:49:20] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10aborrero) [13:49:37] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10aborrero) p:05Triage→03Normal [13:49:39] jynus: I just created https://phabricator.wikimedia.org/T218029 [13:50:03] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10aborrero) [13:59:41] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10Marostegui) So, just to clarify, this is having a similar independent setup like we have in eqiad:m5 but in codfw? [14:11:00] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10aborrero) >>! In T218029#5015217, @Marostegui wrote: > So, just to clarify, this is having a similar independent setup like we have in eqiad:m5 b... [14:15:16] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10Marostegui) Yes, it was kinda to understand the setup. I guess it would need to be renamed to m6 or whatever we decide. So there are a few thing... [14:20:03] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10jcrespo) @Marostegui I mentioned the same issues at T217891#5015199 [14:21:46] 10DBA, 10cloud-services-team (Kanban): CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts - https://phabricator.wikimedia.org/T218029 (10Marostegui) Ah - thanks, I didn't see that, good that we are aligned though :) [15:23:52] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10Marostegui) So, I am going to use this task to report what I have seen whilst testing https://gerrit.wikimedia.org/r/494899: I might be doing crazy things or... [15:55:23] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10jcrespo) Known bugs right now (CC @Marostegui): * All transfers use port 4444, so not good for concurrency (either it fails or corruption is cr... [16:26:04] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo) > And what I get is that I don't really know what happened as there is no usage or trace of error. You just asked for 0 backups, and the application...