[00:03:49] <wikibugs>	 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Dzahn) ` [phab1001:/home/aklapper] $ mysql ERROR 2002 (HY000): Can't connect to local MySQL s...
[00:05:47] <wikibugs>	 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Dzahn) You can switch to the other databases we idenitified with "use" after you connected. F...
[00:09:08] <wikibugs>	 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Dzahn) 05Open→03Resolved Let me know if this works for you.
[03:36:13] <wikibugs>	 10DBA, 10Operations, 10SRE-Access-Requests: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Aklapper) @dzahn: Uh yay! Works! Thanks so much! <3
[05:48:18] <wikibugs>	 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui)
[05:51:31] <wikibugs>	 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui)
[06:24:42] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115 (10Marostegui) a:03Marostegui
[06:25:09] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114 (10Marostegui) a:03Marostegui
[06:25:25] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for szywiki - https://phabricator.wikimedia.org/T237373 (10Marostegui) a:03Marostegui
[06:25:45] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10Marostegui) a:03Marostegui
[06:26:04] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10Marostegui) a:03Marostegui
[06:30:16] <marostegui>	 !log Sanitize shywiktionary gcrwiki szywiki minwiktionary gewikimedia on db2094:3313 T238115 T238114 T237373 T238522 T236404
[06:30:16] <stashbot>	 marostegui: Not expecting to hear !log here
[06:30:16] <stashbot>	 T236404: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404
[06:30:17] <stashbot>	 T237373: Prepare and check storage layer for szywiki - https://phabricator.wikimedia.org/T237373
[06:30:17] <stashbot>	 T238115: Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115
[06:30:17] <stashbot>	 T238114: Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114
[06:30:18] <stashbot>	 T238522: Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522
[06:30:23] <marostegui>	 gah
[06:49:57] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10Marostegui) a:05Marostegui→03None Database sanitized, `_p` database created and grants done - #cloud-services-team please proceed with the views on the 4 hosts. ` root...
[06:51:04] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10Marostegui) a:05Marostegui→03None Database sanitized, `_p` database created and grants done - #cloud-services-team please proceed with t...
[06:53:13] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for szywiki - https://phabricator.wikimedia.org/T237373 (10Marostegui) a:05Marostegui→03None Database sanitized, `_p` database created and grants done - #cloud-services-team please proceed with the views on the 4 host...
[06:54:55] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114 (10Marostegui) a:05Marostegui→03None Database sanitized, `_p` database created and grants done - #cloud-services-team please proceed with the vie...
[06:56:17] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115 (10Marostegui) a:05Marostegui→03None Database sanitized, `_p` database created and grants done - #cloud-services-team please proceed with t...
[06:56:32] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10ArielGlenn) >>! In T68025#5678460, @Ladsgroup wrote: ... >  > I would say we should add size and not just the number of rows. There...
[06:56:44] <marostegui>	 !log Repool labsdb1009
[06:56:44] <stashbot>	 marostegui: Not expecting to hear !log here
[06:57:03] <marostegui>	 I guess I am not awake yet
[08:18:50] <wikibugs>	 10DBA: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui) db2133 is now m2 codfw master db2067 will be decommissioned in a few days ` mysql --skip-ssl -hm2-master.codfw.wmnet -e "select @@hostname" +------------+ | @@hostname | +------------+ | db2133     | +------------+ `
[08:18:58] <wikibugs>	 10DBA: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui)
[08:22:11] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10elukey) @jbond thanks a lot for the ping, in my todo list there is the action item of adding cert expiry checks to all the hadoop nodes (haven't had the time to do...
[08:27:36] <wikibugs>	 10DBA, 10Patch-For-Review: Decommission db2067.codfw.wmnet - https://phabricator.wikimedia.org/T233185 (10Marostegui)
[08:28:13] <wikibugs>	 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui)
[08:49:40] <jynus>	 https://logstash.wikimedia.org/goto/3a3b0f169122a34573d222693544c421
[08:51:14] <jynus>	 db1101 and dbproxy1008 seem to be the worst offenders
[08:54:40] <marostegui>	 yeah, I saw that today in the morning
[08:54:47] <marostegui>	 I wonder why db1101
[09:15:32] <wikibugs>	 10DBA, 10Patch-For-Review: decommission db1067.eqiad.wmnet - https://phabricator.wikimedia.org/T238297 (10Marostegui)
[09:16:22] <jynus>	 did you do a table check/copy of db1067?
[09:16:34] <marostegui>	 yes, I did a compare
[09:38:19] <wikibugs>	 10DBA, 10Operations, 10User-notice: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) All the hosts in s7 have been upgraded to 10.1.43 (including labs and sanitariums)
[10:10:45] <jynus>	 akosiaris: lol at job label https://grafana.wikimedia.org/d/413r2vbWk/bacula
[10:11:13] <akosiaris>	 heh
[10:11:20] <akosiaris>	 could have been worse :P
[10:11:43] <akosiaris>	 pretty long strings though indeed
[10:11:51] <jynus>	 it doesn't work yet BTW, don't get too excited
[10:12:12] * akosiaris writing slides, no excitement here right now
[11:03:11] <godog>	 jynus: left a couple of comments on the exporter, lgtm overall
[11:03:19] <godog>	 bacula-exporter that is
[11:04:59] <jynus>	 ok
[11:05:29] <jynus>	 I don't see them, which patch?
[11:05:58] <godog>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/551145/9/modules/bacula/files/prometheus-bacula-exporter.py
[11:07:37] <jynus>	 thanks
[11:08:04] <jynus>	 apparently it didn't notify me on email
[11:08:18] <jynus>	 now it just arrived
[11:08:22] <godog>	 I don't know bacula well enough, although I'm curious about metrics like pool_id and file_set_id how they could be used ?
[11:08:38] <jynus>	 yeah, there is some work to do
[11:09:03] <jynus>	 pool_id is interesting becaues one may want to filter by those
[11:09:34] <jynus>	 but that is why I wanted to have something to see what will be useful
[11:09:59] <jynus>	 as I mentioned, we will add and remove more metrics
[11:10:30] <godog>	 yep sounds good! thanks for working on that
[11:10:49] <jynus>	 it is not only due to the metrics
[11:11:10] <jynus>	 it is because I don't know what things will break or be used for debugging
[11:11:16] <jynus>	 on bacula itself
[11:11:40] <jynus>	 godog: quick question, do you know what could be happening here: https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1&panelId=10&fullscreen&edit
[11:12:31] <jynus>	 I get [] as a result
[11:13:18] <godog>	 jynus: mhh I get a result changing = to =~ but I'm not exactly sure why now
[11:14:25] <jynus>	 don't worry too much, I will debug it myself, just asking in case it was an obvious mistake
[11:14:26] <godog>	 brb
[11:19:20] <jynus>	 I think something is adding "%5C%5C" (\) to the query
[11:19:41] <jynus>	 to escape .
[11:21:08] <jynus>	 https://github.com/grafana/grafana/issues/6888
[11:31:36] <godog>	 interesting! TIL
[11:32:31] <jynus>	 it works now https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1
[11:32:33] <jynus>	 sorta
[11:35:24] <godog>	 heheh indeed
[11:35:49] <jynus>	 the data is correct, but I have no idea why it is converting a recent timestamp to a 1970 day
[11:36:00] <jynus>	 it is interpreting seconds as days maybe?
[11:37:07] <jynus>	 https://community.grafana.com/t/problem-with-epoch-timestamps-to-human-readable-date-conversion/3576/3
[11:37:28] <jynus>	 x1000
[11:37:43] <godog>	 I don't know, I think though that using time() - <metric> would be more readable
[11:37:55] <jynus>	 I didn't get that
[11:38:04] <jynus>	 if you mean "X days ago"
[11:38:13] <godog>	 yeah that
[11:38:14] <jynus>	 I plan to do that, but that is a different issue
[11:38:23] <jynus>	 it now says "50 years ago"
[11:38:32] <jynus>	 so it is easier to debug this way (temporarelly)
[11:38:50] <jynus>	 it is the value interpretation that is problemtic, not the formatting :-D
[11:40:46] <jynus>	 should be ok now https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1
[11:40:54] <jynus>	 these 2 issues were weird
[11:41:02] <jynus>	 but things should be ok now
[11:41:16] <jynus>	 I will now work on improving the exporter
[11:43:04] <godog>	 ah ok got what you meant now
[14:18:04] <marostegui>	 Amir1: Just finished watching your presentation about wikidata, very interesting and useful to understand things a lot better :)
[14:18:57] <Amir1>	 marostegui: thanks, I hope it will be useful for you ^_^
[14:19:03] <marostegui>	 Definitely :)
[15:54:54] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10JHedden) 05Open→03Resolved a:03JHedden  `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --database gewikimedia --debug $ sud...
[15:56:06] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10JHedden) 05Open→03Resolved a:03JHedden  `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --dat...
[16:01:04] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for szywiki - https://phabricator.wikimedia.org/T237373 (10JHedden) 05Open→03Resolved a:03JHedden  `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --database szywiki --debug...
[16:03:42] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114 (10JHedden) 05Open→03Resolved a:03JHedden  `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --database...
[16:04:55] <wikibugs>	 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115 (10JHedden) 05Open→03Resolved a:03JHedden `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --data...