[02:06:12] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @Marostegui Pinging for review of these two files, https://phabricator.wikimedia.org/diffusion/EJ... [06:49:33] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [07:06:24] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [07:19:07] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [07:27:20] 10DBA, 10MediaWiki-General-or-Unknown, 10MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): [Bug] Update old nonuniformly distributed page_random values - https://phabricator.wikimedia.org/T208909 (10Tbayer) 05Open>03Resolved... [07:41:51] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [08:02:52] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [08:03:20] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) 05Open>03Resolved This is all done [08:04:26] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) {T203709} is now done, so I am going to proceed with this one //cc @Ladsgroup [08:12:49] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:15:50] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:18:48] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:20:24] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:23:25] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:24:47] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) [08:25:34] 10DBA, 10Patch-For-Review: Drop ct_ indexes on change_tag - https://phabricator.wikimedia.org/T205913 (10Marostegui) 05Open>03Resolved This is all done [08:25:38] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [08:27:43] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) a:03Marostegui [08:32:04] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [08:54:14] 10DBA, 10User-Banyek: Schema change task for dropping user options - https://phabricator.wikimedia.org/T209458 (10Banyek) [09:12:44] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) s6 progress: [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore2001 [] dbstore1002 [] dbstore1001 [] db2095 [] db2089 [] db2087 [] db2076 [] db2... [09:13:00] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [09:13:26] 10Blocked-on-schema-change, 10DBA, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:14:01] 10DBA, 10User-Banyek: Schema change task for dropping user options - https://phabricator.wikimedia.org/T209458 (10Banyek) 05Open>03Invalid The original ticket was modified instead of this. [09:14:36] 10Blocked-on-schema-change, 10DBA, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:27:06] Amir1: all the clean ups tickets related to change_tag have been done (adding the new indexes and removing the duplicate ones) [09:29:01] 10Blocked-on-schema-change, 10DBA, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:36:24] 10Blocked-on-schema-change, 10DBA, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:37:06] 10Blocked-on-schema-change, 10DBA, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:57:28] 10DBA, 10MediaWiki-General-or-Unknown, 10MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): [Bug] Update old nonuniformly distributed page_random values - https://phabricator.wikimedia.org/T208909 (10phuedx) As promised, here are... [09:58:14] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T202367 (10Banyek) a:03Banyek [10:17:21] marostegui: Thank you so much! [10:21:01] I have bad feelings about the sanitariums. Last week the archive table, now the pagelings [10:21:04] I have bad feelings about the sanitariums. Last week the archive table, now the pagelinks [10:21:32] it could be due to the s8 issue we had [10:21:37] although all the tables were checked [10:21:40] have you found the missing row? [10:21:45] yes [10:21:54] I insert it manually and restart replication [10:22:01] do you want me to double check it? [10:22:11] just to see if we reach to the same conclusion [10:22:19] if you have time, sure [10:22:26] I'd appreciate it [10:22:27] let me check [10:23:34] ### @1=5069396 [10:23:34] ### @2=0 [10:23:34] ### @3='Q639669' [10:23:34] ### @4=0 [10:23:36] that? [10:23:43] yes [10:23:46] ok [10:23:56] how will you insert it? [10:25:00] ```SET SESSION sql_log_bin=0; INSERT INTO wikidatawiki.pagelinks VALUES (### @1=5069396,0,"Q639669",0);``` [10:25:04] meh [10:25:12] that will break labs [10:25:25] wrong paste [10:25:31] SET SESSION sql_log_bin=0; INSERT INTO wikidatawiki.pagelinks VALUES (5069396,0,"Q639669",0); [10:25:33] why [10:25:49] ? [10:26:28] I would assume that row doesn't exist on labs either [10:27:51] well, it worth to check imho [10:27:54] let me [10:28:08] just to make sure [10:28:24] sec. [10:28:27] ok [10:33:34] good that row miss everywehere [10:33:40] so ```INSERT INTO wikidatawiki.pagelinks VALUES (5069396,0,"Q639669",0);``` [10:33:46] with binlog enabled on db1124 [10:33:49] it will replicate [10:33:58] yes [10:34:22] good [10:34:28] Inserting the row now [10:35:03] https://www.irccloud.com/pastebin/sDYkFjlK/ [10:35:21] row is in, replcation started [10:35:25] great [10:35:30] thanks [10:36:42] good [10:36:52] I am going back to schema change [10:36:57] ok [10:40:18] btw. do you have trick for extracting binlog events? I always do ```mysqlbinlog --stop-position --base64-output=decode-rows -v > binlog.out``` and check the last event, but is there any better way? [10:41:02] I normally do that too [10:41:18] I use —start-position though [10:45:25] but how you know what was the prev. one? [10:45:34] I am always planning to create a tool for that [10:45:49] I use the start position as the last one executed by the sql thread [10:46:08] oh, THAT'S the one I was curious for [10:46:09] :) [10:46:19] anyways, I am getting back to the schema change [10:46:32] muy door is here [10:53:45] I downtimed the replication checks for db2046 [10:53:58] ok [11:08:36] I'll execute [11:08:39] ```./software/dbtools/osc_host.sh --host db2046 --dblist mediawiki-config/dblists/s6.dblist --table user --no-replicate "CHANGE COLUMN user_options _user_options_drop blob NOT NULL"``` [11:08:48] from cumin2001.codfw.wmnet [11:10:48] ok [11:11:01] why not the .py version? [11:11:44] because I see a wrong repo [11:11:56] uh? [11:12:54] ```root@cumin2001:/home/banyek# ls -l software/dbtools/osc* [11:12:54] -rwxr-xr-x 1 root root 6604 Nov 14 11:07 software/dbtools/osc_host.sh``` [11:13:13] it is on the wmfmariadbpy I think [11:13:47] that's why I said I am seeing a wrong repo [11:13:54] osc_ [11:15:03] so [11:15:05] ```./wmfmariadbpy/wmfmariadbpy/osc_host.py --host db2046 --dblist mediawiki-config/dblists/s6.dblist --table user --no-replicate "CHANGE COLUMN user_options _user_options_drop blob NOT NULL"``` [11:15:20] ok [11:15:23] looks good [11:15:30] I haven't reviewed the alter syntax though [11:15:34] I assume you did ;) [11:17:08] I did back then [11:17:09] but [11:17:12] ```CHANGE [COLUMN] old_col_name new_col_name column_definition``` [11:17:31] sure, I trust you :) [11:17:34] I was just saying [11:18:05] is there a reason why osc_host.py doestn have execute privilege after checking out the repo? [11:18:26] Don't know, I guess it was uploaded like that [11:18:34] ok [11:18:40] so I am hitting enter in [11:18:40] 3 [11:18:42] 2 [11:18:43] 1 [11:20:06] https://coub.com/view/shdqb [11:20:17] hehe [11:20:21] ```File "./wmfmariadbpy/wmfmariadbpy/osc_host.py", line 20, in [11:20:21] from wmfmariadbpy.WMFMariaDB import WMFMariaDB [11:20:21] ImportError: No module named 'wmfmariadbpy'``` [11:20:32] adding PYTHONLIB [11:20:36] ah, same that happened to you with transfer.py I think? [11:20:43] yes [11:20:47] how did you fix it in the end? [11:21:10] I added PYHTONLIB to the beggining of the command [11:22:39] PYTHONPATH=/home/banyek/wmfmariadbpy/ ./wmfmariadbpy/wmfmariadbpy/osc_host.py --host db2046 --dblist mediawiki-config/dblists/s6.dblist --table user --no-replicate "CHANGE COLUMN user_options _user_options_drop blob NOT NULL" [11:22:53] ```ost : db2046 [11:22:53] Port : 3306 [11:22:53] Databases : ['frwiki', 'jawiki', 'ruwiki'] [11:22:53] Table : user [11:22:53] Alter SQL : CHANGE COLUMN user_options _user_options_drop blob NOT NULL [11:22:54] method : percona [11:22:54] pt dry args : ['--recurse=0', '--set-vars=sql_log_bin=off', '--check-slave-lag=db2046'] [11:22:55] pt args : ['--recurse=0', '--set-vars=sql_log_bin=off', '--check-slave-lag=db2046'] [11:22:55] ddl args : ['SET SESSION innodb_lock_wait_timeout=1;', 'SET SESSION lock_wait_timeout=60;', 'set session sql_log_bin=0;'] [11:22:56] analyze : False [11:22:56] continue? yes/no``` [11:23:04] please use paste for that [11:24:47] https://phabricator.wikimedia.org/P7804 [11:25:30] what problem? [11:26:00] I don't know yet, now I am checking the host [11:26:50] I guess you need FQDN? [11:27:16] let's see [11:27:42] same error [11:28:21] I'll use alter table rename column instead of alter table change column then [11:29:00] is it a syntax issue? [11:29:22] it doesn't said more than in the phaste [11:29:25] paste [11:29:57] with RENAME COLUMN [11:29:59] ```SKIPPING frwiki : user dry-run encountered problems``` [11:30:19] let's see if there's a --verbose switch [11:30:28] there is debug [11:30:33] as far as I remember [11:30:48] yes, --debug [11:30:53] lemme [11:34:20] mmmmrgh [11:34:52] so RENAME COLUMN is non-existent in mariadb [11:35:00] change column is better: [11:35:05] yes, rename doesn't exist [11:35:06] ```Altering `frwiki`.`user`... [11:35:06] --alter appears to rename these columns: [11:35:06] user_options to _user_options_drop [11:35:06] The tool should handle this correctly, but you should test it first because if it fails the renamed columns' data will be lost! Specify --no-check-alter to disable this check and perform the --alter. [11:35:06] `frwiki`.`user` was not altered. [11:35:07] --check-alter failed. [11:35:07] WARNING frwiki : user encountered problems``` [11:35:33] that's why the dry-run stopped ^^ with change column [11:36:12] is there any reason not to continue? [11:36:22] I don't see any [11:37:01] well, I would first try to investigate what the warning is [11:38:34] does it happen with osc_host.sh too? [11:39:36] as I see this is the test itself. It just warns if the rename is not successful the column will be dropped - as we want to drop the column anyway, plus it is actually the test itself, and the host is depooled I think we should proceed [11:39:50] I don't knmow how the .sh worked, I didn't started that change [11:40:09] but why is the test failing? [11:40:57] Maybe try to add a dummy column and then drop it, and see if it complains to [11:41:01] *too [11:41:20] https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html [11:41:40] ```Column renames [11:41:40] In previous versions of the tool, renaming a column with CHANGE COLUMN name new_name would lead to that column’s data being lost. The tool now parses the alter statement and tries to catch these cases, so the renamed columns should have the same data as the originals. However, the code that does this is not a full-blown SQL parser, so you should first run the tool with --dry-run and --print and verify that it detects the renamed [11:41:40] columns correctly.``` [11:42:03] it warns because we do a rename [11:42:27] let's do that then, and check the —print [11:43:15] ok, continuing, and I put the output to a paste [11:43:32] So you'll do a —dry-run —print ? [11:44:47] yes [11:44:57] ok, let's see what that shows [11:47:38] https://phabricator.wikimedia.org/P7804 [11:47:43] I added as a comment [11:48:25] note to self: I add the --print option for dry-run if --debug specifed to osc_host.py [11:48:26] that is wrong [11:48:32] we don't want to use triggers [11:48:39] use --method=ddl [11:49:40] I need to update the docs [11:49:43] now I run it [11:49:55] try the dry-run again [11:49:59] with the method=ddl [11:51:39] with method=ddl is there dry-run? iirc that was the option for pt-osc [11:51:40] ? [11:51:50] don't know, don't remember :) [11:51:58] ah, ok [11:52:05] don't know what the wrapper does, can't remember [11:52:21] it asks me if continue, so I let the tool do whatever it has to do [11:52:27] sure [11:53:15] see paste [11:53:33] rename was successful [11:53:53] replication is running [11:53:53] check it to make sure it was actually good [11:54:07] the table definition changed, the replication runs [11:54:14] good [11:56:01] can you review my patch btw? I am blocked on your review :) [11:56:48] sure sure [11:56:52] thanks [11:59:53] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [12:01:36] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) The testing in db2046 was initiated with : `PYTHONPATH=/home/banyek/wmfmariadbpy/ ./wmfmariadbpy/wmfmariadbpy/osc_host.... [12:42:22] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) [12:42:40] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) p:05Triage>03High [12:47:00] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) This is the current list of conn procs in m5-master as pointed by @Volans ` MariaDB [(none)]> select LEFT(HOST, LOCATE(':',HOST) - 1) as h, count(*) as count fr... [12:52:27] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) We have several log messages like the following in `/var/log/nova/nova-api.log` in labnet1001: `lines=10 2018-11-14 12:06:19.549 29112 ERROR nova.api.openstack.e... [13:11:17] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) This can be found in /var/log/nova/nova-api.log in labnet1002: ` 2018-11-14 12:07:30.685 16639 ERROR oslo_service.service DBConnectionError: (pymysql.err.Operati... [13:22:02] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wikidata-tech-focus: wikibase: synchronize schema on production with what is created on install - https://phabricator.wikimedia.org/T85414 (10Addshore) [13:22:16] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wikidata-tech-focus: wikibase: synchronize schema on production with what is created on install - https://phabricator.wikimedia.org/T85414 (10Addshore) [13:22:49] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wikidata-tech-focus, 10User-Addshore: wikibase: synchronize schema on production with what is created on install - https://phabricator.wikimedia.org/T85414 (10Addshore) [13:22:55] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wikidata-tech-focus, 10User-Addshore: wikibase: synchronize schema on production with what is created on install - https://phabricator.wikimedia.org/T85414 (10Addshore) 05Open>03Resolved a:03Addshore [13:26:49] dbproxies will replace the old ones in order, or there's a different mapping? [13:28:00] I mean dbproxy1001 -> dbproxy1012; 1002 -> 1013 etc. [13:29:24] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) The number of connections to m5-master from cloudcontrol1003 is very high: ` aborrero@cloudcontrol1003:~ $ sudo netstat -putan | grep 10.64.16.79 | wc -l 335 ` (... [13:36:20] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10GTirloni) ` cloudcontrol1003# lsof -i -a -n | grep 10.64.16.79 | awk '{ print $1 }' | sort | uniq -c |sort -rn 120 neutron-s 87 keystone- 61 nova-api 27... [13:39:29] 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (101997kB) 05Open>03stalled [13:43:53] 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (101997kB) Stalled until waiting time for usurpation is over. [13:44:57] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10aborrero) p:05Normal>03High [13:45:36] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10aborrero) Bump. I would like to see if we have any chances on moving forward with this. [13:49:50] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) See T202889#4541135 [13:50:17] 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (10Marostegui) Please coordinate with @Banyek for this. [13:52:12] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10aborrero) >>! In T202889#4746340, @Marostegui wrote: > See T202889#4541135 What do you prefer? What would you recommend? Also, @bd808 what do you think? [14:18:38] marostegui: the dbproxy1005 error was caused the same reason arturo mentioned (db1073 was out of connections) do I need anything else than reload haproxy to recover it? [14:19:29] good that m5 doesn't use a proxy... [14:19:52] so yes [14:19:53] tx [14:58:55] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) I have added pc2010 as spare host with the following line on db-codfw.php - we can change it if we want t... [14:59:17] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [14:59:23] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10Banyek) I don't see any reason for not to increase the limit, 500 conections are not too much, but actually the most of the threads are idle: (are in sleep state) `... [15:04:23] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [15:05:14] I drop the renamed column on db2046, (the replication is not broken) and prepare depooling db1088 [15:05:22] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) @Cmjohnson any ETA to get these racked&installed? Thanks [15:05:23] good [15:05:53] about proxies: the new ones will be 1:1 replacements or there's a different mapping? [15:06:13] in other words can I use hieradata of dbproxy1001 to dbproxy1012 ? [15:06:24] it will be a 1:1 [15:06:28] good [15:06:30] tx [15:12:49] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10Marostegui) >>! In T209480#4746549, @Banyek wrote: > I don't see any reason for not to increase the limit, 500 conections are not too much, but actually the most of the thr... [15:15:35] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) dropping renamed coumn on db2046 `PYTHONPATH=/home/banyek/wmfmariadbpy/ ./wmfmariadbpy/wmfmariadbpy/osc_host.py --method... [15:26:47] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) >>! In T202889#4746346, @aborrero wrote: >>>! In T202889#4746340, @Marostegui wrote: >> See T202889#4541135 > > What do you prefer? What would you reco... [15:36:00] marostegui: I need a +1 on this too: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473536/ [15:36:13] (repooling db2046) [15:36:43] For reverts we don't normally +1 :-) [15:36:51] oh, good [15:37:05] I repool it now then [15:37:33] I can start working on db1088 too, but at 5 I have to go for a travel training [15:37:47] ok [15:37:55] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10Bstorm) There is a cap on the user connections (the user that eats connections being OpenStack aka Cloud VPS). It just has burst capabilities and can briefly go over what... [15:38:04] so I think after I repooled db2046 I'll prepare the proxies, and leave 1088 for tomorrow as I don't want to leave it halfly done [15:38:13] ok [15:38:21] 👍 [15:59:26] I leave for the training now [17:38:50] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [17:39:50] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [17:42:31] 10DBA, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) [17:43:00] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) [17:56:42] 10DBA: replication broken on db1124 - https://phabricator.wikimedia.org/T209521 (10Banyek) [17:56:51] 10DBA: replication broken on db1124 - https://phabricator.wikimedia.org/T209521 (10Banyek) 05Open>03Resolved [17:57:15] I call this a day. [17:58:27] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) @Halfak labsdb1004/5 would affect wikilabels. We may just do reboots in place like last time due to the tables that don't replicate per: https://wikitech.wikimedia.o... [23:39:34] 10DBA, 10JADE, 10Operations, 10Epic, and 2 others: [Epic] Extension:JADE scalability concerns - https://phabricator.wikimedia.org/T196547 (10awight) 05Open>03Resolved This was addressed for now, by an agreement between our team and SRE to not install JADE on wikis with revision table size >= 100GB. Th...