[08:15:10] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) No trace of errors on logtash for this wiki since 11th Dec. So I am going to proceed dropping these tables
[08:22:58] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) I have left a temporary backup at: ` root@db1078:/srv/tmp/T209761 `
[08:34:02] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui)
[08:34:08] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) 05Open>03Resolved Tables are gone: ` root@db1075.eqiad.wmnet[srwikinews]> show tables like '%flagged%';...
[09:42:20] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (10Marostegui) @1997kB can we do this tomorrow?
[09:43:33] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (101997kB) Sure, what time?
[09:44:28] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (10Marostegui) Let's do it at 10 UTC, would that work for you? //cc @Banyek
[09:44:39] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (101997kB) 05stalled>03Open
[09:46:14] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (101997kB) Perfect time for me.
[09:46:45] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Massimo Telò → Teseo: supervision needed - https://phabricator.wikimedia.org/T209488 (10Marostegui) Cool - ping @Banyek on this task before you are ready to start! Thanks!
[10:00:34] <Hauskatze>	 marostegui: hola - can you `show tables;` on prod for me and see if there's one called globalblocking or similar?
[10:00:45] <Hauskatze>	 it's missing on cloud replicas
[10:00:47] <marostegui>	 Hauskatze: on which wiki?
[10:01:13] <Hauskatze>	 un minuto, teléfono
[10:01:16] <marostegui>	 :)
[10:02:25] <marostegui>	 Hauskatze: I am checking replication filters, there is one table called globalnames which we don't replicate to cloud replicas
[10:03:02] <Hauskatze>	 marostegui: back
[10:03:06] <Hauskatze>	 for example, eswikibooks
[10:03:24] <Hauskatze>	 I'm looking for the table used at https://www.mediawiki.org/wiki/Extension:GlobalBlocking
[10:03:29] <Hauskatze>	 which is public information
[10:03:32] <marostegui>	 global_block_whitelist?
[10:03:36] <marostegui>	 maybe that?
[10:03:38] <Hauskatze>	 nope
[10:03:51] <Hauskatze>	 that is the local whitelist for global blocks
[10:03:56] <marostegui>	 That is the only one we have by the name glob%
[10:04:06] <Hauskatze>	 I am looking for the table where global blocks are stored
[10:04:17] <marostegui>	 ipblocks?
[10:04:28] <marostegui>	 https://phabricator.wikimedia.org/P7912
[10:04:50] <Hauskatze>	 ipblocks are local blocks only afaik
[10:05:02] <marostegui>	 Let me check centralauth
[10:05:22] <marostegui>	 Yeah
[10:05:25] <marostegui>	 It is on centralauth
[10:05:54] <Hauskatze>	 so centralauth_p for me I guess?
[10:05:55] <marostegui>	 globalblocks is on centralauth
[10:05:56] <marostegui>	 yep
[10:05:58] <marostegui>	 it is there
[10:06:00] <marostegui>	 I just checked
[10:06:06] <Hauskatze>	 perfect, thanks
[10:06:10] <marostegui>	 de nada!
[10:06:34] <Hauskatze>	 what's 'cur' and 'hidden' on eswikibooks?
[10:11:21] <marostegui>	 No idea :)
[10:11:31] <marostegui>	 hidden looks empty
[10:11:54] <marostegui>	 And cur looks unused
[10:11:57] <marostegui>	 not written since 2015
[10:12:06] <marostegui>	 root@db1075:/srv/sqldata/eswikibooks# ls -lh cur.ibd hidden.ibd
[10:12:06] <marostegui>	 -rw-rw---- 1 mysql mysql  14M Nov 18  2015 cur.ibd
[10:12:07] <marostegui>	 -rw-rw---- 1 mysql mysql 176K Nov 18  2015 hidden.ibd
[10:12:16] <marostegui>	 I guess leftovers?
[10:12:54] <hoo>	 cur was a table containing (just) the current page texts for all pages
[10:13:05] <marostegui>	 hey hoo o/ :)
[10:13:12] <hoo>	 Unused since forever (way before 2015, I think)
[10:13:26] <Hauskatze>	 drop it?
[10:13:37] <Hauskatze>	 maybe we can have a task for it
[10:13:59] <marostegui>	 https://phabricator.wikimedia.org/T54921
[10:14:11] <marostegui>	 It says it needs audit?
[10:14:48] <Hauskatze>	 somewhat, I'm not sure I understand Tim's reasoning there
[10:14:52] <Hauskatze>	 if it's empty...
[10:15:04] <marostegui>	 might not be empty everywhere
[10:15:10] <marostegui>	 maybe ask him on that task?
[10:15:22] <Hauskatze>	 perhaps
[10:15:46] <Hauskatze>	 but I assume he knows better
[10:16:49] <marostegui>	 hidden also appears there
[10:17:48] <Hauskatze>	 part of the old Oversight extension
[10:18:00] <Hauskatze>	 probably not needed anymore but dunno
[10:18:58] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) @Krenair I was talking with @MarcoAurelio about the `hidden` table on eswikibooks and we saw it is listed here - can it be dropped everywhere?
[10:31:36] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Krenair) >>! In T54921#4819782, @Marostegui wrote: > @Krenair I was talking with @MarcoAurelio about the `hidden` table on eswikibooks and we saw it is...
[10:31:57] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Krenair)
[10:49:47] <moritzm>	 hi, ok to reboot the dbmonitor hosts? (to pick up the SSBD-enabled qemu) if that currently messes with any DBA maintenance work, I can also do it at a different time
[10:49:59] <marostegui>	 moritzm: go for it!
[10:50:19] <moritzm>	 ack
[10:56:14] <moritzm>	 both rebooted and confirmed working fine
[10:56:46] <marostegui>	 \o/
[11:47:26] <_joe_>	 FWIW, we just had another peak in the read_rnd_next similar to yesterday
[11:47:32] <_joe_>	 https://grafana.wikimedia.org/d/000000273/mysql?panelId=3&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1084&var-port=9104&kiosk&refresh=10s&from=now-3h&to=now
[11:47:37] <_joe_>	 commons again
[11:47:56] <_joe_>	 we could use it to confirm that query you found yesterday is really the issue marostegui
[11:48:04] <marostegui>	 I asked Amir1 to review the patch anomie put together
[11:48:12] <marostegui>	 It is a one line fix, that we should merge asap
[11:49:22] <marostegui>	 It is the same query, _joe_ just checked it - let me update the task
[11:49:58] <Amir1>	 marostegui: Hey, Sorry for not responding earlier (guess who just woke up). I will get to it asap. It's a one line change but it has lots of affects, I don't want it to leave the production in a more broken state
[11:50:27] <Amir1>	 I need dig a little bit in the code to make sure it works as expected 
[11:50:28] <Hauskatze>	 morning Amir1 :)
[11:50:38] <marostegui>	 Amir1: Sure! Sorry for getting you involved! I am going to update the task and get others to see if they can help out with the review!
[11:50:48] <Amir1>	 Technically it's afternoon now :D
[11:50:52] <marostegui>	 haha
[11:50:58] <_joe_>	 Amir1: you need to sync with your TZ :P
[11:51:14] <Amir1>	 TZ needs to sync with me :D
[11:51:32] <_joe_>	 Amir1: if you want, marostegui and I can phone you when we start to work
[11:51:35] <_joe_>	 to wake you up
[11:51:36] <Amir1>	 I grab coffee and check it and hopefully gets merged before 2
[11:51:40] <_joe_>	 around 7 AM :P
[11:51:49] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Marostegui) We had another spike on commons, same query.  https://grafana.wikimedia.org/d/000000273/mysql?panelId=3&fullscreen&orgId=1&var-dc=e...
[11:51:49] <marostegui>	 Amir1: <3
[11:52:10] <Amir1>	 I would answer you, because that's around the time I go to sleep
[11:52:18] <Amir1>	 _joe_: ^
[11:52:24] <_joe_>	 lol
[11:52:38] <marostegui>	 Hahaha
[11:59:46] <Amir1>	 brb
[12:41:51] <Amir1>	 back now, let's check the patch
[12:42:02] <marostegui>	 thanks!
[12:48:06] <Amir1>	 marostegui: +2'd it. We can backport it to prevent more polar bears from dying 
[12:48:41] <marostegui>	 Amir1: Awesome!!!!!
[12:48:47] <marostegui>	 Thanks a lot!
[12:48:53] <marostegui>	 Can you maybe comment on the task?
[12:50:40] <Amir1>	 done
[12:50:41] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Ladsgroup) +2'd, I leave it to our DBAs if they think this needs to be backported or we should wait until wmf.9. If you think this needs backpo...
[12:51:26] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Marostegui) >>! In T211804#4820148, @Ladsgroup wrote: > +2'd, I leave it to our DBAs if they think this needs to be backported or we should wai...
[13:41:19] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) >>! In T207258#4804302, @Cmjohnson wrote: > @marostegui and all, >  > the system board that was replaced yesterday was faulty. Showing errors on DIMM slots B4 and B1.  After swapping...
[14:10:44] <Amir1>	 marostegui: The next available SWAT is at 8pm, I'm not sure if I would be around by then  but will give it a try (I have other stuff to deploy too)
[14:11:20] <marostegui>	 Amir1: Sure! Thank you :)
[14:11:25] <marostegui>	 Thanks for the backport :)
[14:15:18] <Amir1>	 no worries
[15:13:33] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10MW-1.33-notes (1.33.0-wmf.9; 2018-12-18), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Anomie) 05Open>03Resolved a:03Anomie Worked around and backported
[15:14:06] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10MW-1.33-notes (1.33.0-wmf.9; 2018-12-18), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Marostegui) Thank you so much @Anomie and @Ladsgroup!!
[16:18:56] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) no ETA
[18:22:50] <wikibugs>	 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10RobH) 05Open>03stalled a:05RobH>03Papaul @papaul,  We will be returning these to Farnam sometime this o...
[18:43:09] <wikibugs>	 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul) 05stalled>03Resolved This can be resolved then since i am done with it .
[20:09:04] <wikibugs>	 10DBA, 10Core Platform Team Kanban (Done with CPT), 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10CCicalese_WMF)
[21:40:04] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) >>! In T209761#4819586, @Marostegui wrote: > I have left a temporary backup at: > ` > root@db1078:/srv/tmp...
[21:46:48] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) 05Resolved→03Open I still can find these tables at rwikinews.analytics.db.svc.eqiad.wmflabs:   ` zoran...
[21:46:51] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Zoranzoki21)
[21:52:03] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Krenair) Looks like the tables are gone, the views just need dropping: ` MariaDB [srwikinews_p]> select 1 from flaggedi...
[21:58:53] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) I still see tables at analytics.db.svc.eqiad.wmflabs  {F27541389}
[22:03:49] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Krenair) What you are seeing there is views.
[22:05:47] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Zoranzoki21)
[22:05:52] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) 05Open→03Resolved >>! In T209761#4822217, @Krenair wrote: > What you are seeing there is views.  I rol...
[22:06:27] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Krenair) Well, ideally relevant views would be dropped within such tasks.
[22:07:30] <wikibugs>	 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Urbanecm) i feel prod and cloud is quite relevant, created T211939, let's do it there.
[22:11:26] <wikibugs>	 10DBA, 10Data-Services, 10User-Zoranzoki21: Drop several views from srwikinews - https://phabricator.wikimedia.org/T211939 (10bd808) >>! In T209251#4741279, @Marostegui wrote: > If the extension is disabled and it is decided it won't be enabled again, there is no need to keep the tables and we should clean t...