[05:53:22] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [06:18:38] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [07:14:29] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) [07:44:02] 10DBA, 10Patch-For-Review: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui) db2132 is now m1 codfw master. db2062 will be decommissioned ` root@cumin1001:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hm1-master.codfw.wmnet -e "select @@hostname" +------------+ | @... [07:44:17] 10DBA, 10Patch-For-Review: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui) [08:02:13] 10DBA: Decommission db2061.codfw.wmnet - https://phabricator.wikimedia.org/T238526 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db2061.codfw.wmnet` - db2061.codfw.wmnet (**PASS**) - Downtimed host on Icinga - Downtimed management interface on Icinga... [08:05:43] 10DBA: Decommission db2061.codfw.wmnet - https://phabricator.wikimedia.org/T238526 (10Marostegui) [08:16:25] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [08:17:43] 10DBA: Decommission db2062.codfw.wmnet - https://phabricator.wikimedia.org/T238726 (10Marostegui) [08:18:04] 10DBA: Decommission db2062.codfw.wmnet - https://phabricator.wikimedia.org/T238726 (10Marostegui) p:05Triage→03Normal [08:18:31] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [08:19:44] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [09:25:07] 10DBA: Decommission db2067.codfw.wmnet - https://phabricator.wikimedia.org/T233185 (10Marostegui) 05Stalled→03Open [09:25:09] 10DBA: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui) [09:25:12] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [09:46:03] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [09:50:29] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [09:59:25] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [10:15:24] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [10:28:04] jynus: I want to upgrade db1116, nothing seems to be running on the host and/or on backups table on zarcillo. Can I proceed? [10:30:07] sure [10:30:13] ok, thanks [10:37:25] all done [10:37:56] thanks [12:11:09] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore) I just created a basic dashboard with these metrics. https://grafana.wikimedia.org/d/pwq8ZIxWk/large-site-db-tables Is t... [13:47:42] 10DBA, 10Patch-For-Review: Decommission db2062.codfw.wmnet - https://phabricator.wikimedia.org/T238726 (10Marostegui) [15:04:14] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Ladsgroup) >>! In T68025#5678019, @Addshore wrote: > I just created a basic dashboard with these metrics. > https://grafana.wikimed... [15:27:51] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Marostegui) Thanks @Addshore! I think the size as @Ladsgroup points out could be pretty cool. Not sure if we should add totals and/... [15:28:47] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [16:10:55] there is a 2.5x increase in enwiki rows read since 15:40 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&panelId=8&fullscreen&from=1574244647180&to=1574266247180&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All [16:11:48] on SAL there is only a change by marko but it says beta-only [16:11:58] apart from that 15:36 by addshore but doesn't look related [16:12:11] * addshore reads up [16:12:27] the increase is pretty much everywhere [16:12:36] 15:36 shouldn't have anything to do with enwiki [16:12:46] yeah, there is a general increase everywhere [16:12:56] that change did cause edit rate on wikidata to drop (which the query service catches up) [16:13:08] *while [16:13:34] Amir1: ^^ (fyi, don't think it is us though) [16:13:51] I cannot see that increase on the enwiki slaves I am checking individually [16:16:12] it looks like it is vslow slave on enwiki [16:16:16] dumps maybe? [16:16:44] yes [16:16:46] it is dumps [16:17:11] https://grafana.wikimedia.org/d/000000273/mysql?panelId=3&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1106&var-port=9104&from=now-24h&to=now [16:17:33] cool then [16:17:38] thanks for catching that [16:17:51] * marostegui goes back to the network talk [17:07:52] 10DBA, 10Analytics: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10Ottomata) [17:09:41] 10DBA, 10Analytics: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10Ottomata) T231858 is done so this is unblocked, woohoo! [17:13:38] jynus: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552027/4..7 says no diff [17:17:03] oh [17:17:26] I must have done something wrong [17:18:42] mm, 6 was fixed, I must have reviewed an old version by accident when doing some next on the dependency chain [17:21:27] yeah, the next change hadn't been rebased [17:23:40] now 6 and 8 are the same https://gerrit.wikimedia.org/r/c/operations/puppet/+/552027/6..8 [17:23:45] which means the patch is ok [17:23:54] https://gerrit.wikimedia.org/r/c/operations/puppet/+/552027/8 [17:24:01] well, not sure if ok, but as intended [17:24:06] 0:-) [17:24:48] and I will rebase the next in chain so it doesn't happen again [17:25:27] heh, I like how I was faster than CI [17:25:38] even though I +1ed after at least 2 mins [17:26:39] so my own phylosophy about patches is: is this better than what we have now? does it add risks? no, then +1 [17:26:51] we can always make it better at a latter time [17:27:42] otherwise perfect gets in the way of done [17:28:43] I will not deploy today however, as I am likely to have problems with templates hving small problems or something else, will deploy tomorrow [21:35:09] 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10MarcoAurelio) Hi there. Wiki database just created. Regards. [21:50:41] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10Urbanecm) >>! In T238522#5670499, @Marostegui wrote: > Let us know when the database is created so we can sanitize it before sending it to t... [23:16:57] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for szywiki - https://phabricator.wikimedia.org/T237373 (10Ladsgroup) The wiki has been created. [23:17:14] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114 (10Ladsgroup) The wiki has been created [23:17:23] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115 (10Ladsgroup) The Wiki has been created.