[02:35:48] 10DBA, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 07RfC: change_tag table needs redesign - https://phabricator.wikimedia.org/T164167#3224657 (10TTO) [02:46:49] 10DBA, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 07RfC: change_tag table needs redesign - https://phabricator.wikimedia.org/T164167#3224201 (10TTO) > It uses varchar instead of being a foreign key to another table or at least it's not a hash for faster lookup I assume the reason it (and tag_summary... [08:48:07] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3224731 (10waldyrious) >>! In T126832#3224558, @jcrespo wrote: > waldyrious- no data existe live anymore of the old wiki (only temporarily on backups), not even on labs.... [09:02:22] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3224744 (10Dereckson) The MediaWiki developers position not to implement ACL in namespaces, pages, group of pages, etc. is still the current doctrine. You took a risk with... [11:15:07] 10DBA: Convert unique keys into primary keys for some wiki tables on s6-eqiad - https://phabricator.wikimedia.org/T163979#3224846 (10jcrespo) [11:15:25] 07Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s3-eqiad - https://phabricator.wikimedia.org/T163912#3224847 (10jcrespo) [11:15:58] 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3224848 (10jcrespo) [12:37:58] 10DBA, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 07RfC: change_tag table needs redesign - https://phabricator.wikimedia.org/T164167#3224901 (10Ladsgroup) >>! In T164167#3224664, @TTO wrote: > I assume the reason it (and tag_summary) were written using varchar tag names as identifiers was to avoid h... [12:59:38] 10DBA, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 07RfC: change_tag table needs redesign - https://phabricator.wikimedia.org/T164167#3224918 (10jcrespo) > JOIN is cheap if the proper index is in place what is expensive is slow lookups like this. I kind of agree with that statement in general. There... [13:29:25] 10DBA, 07Epic, 13Patch-For-Review, 05codfw-rollout: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) - https://phabricator.wikimedia.org/T155099#3224953 (10jcrespo) [13:29:27] 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3224952 (10jcrespo) [13:37:13] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3224974 (10jcrespo) [13:42:22] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3224978 (10jcrespo) https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-cli... [15:06:47] 10DBA, 06Operations, 10ops-eqiad: Reset db1070 idrac - https://phabricator.wikimedia.org/T160392#3225192 (10Cmjohnson) @marostegui I am not sure what to make of this...i know several servers have this issue but on the server itself ipmi works fine. [15:14:12] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225208 (10Cmjohnson) This is what I found at http://www.dell.com/support/article/us/en/19/SLN285596/drac---how-to-set-fan-speed-offset-values-in-idrac7-witho... [15:32:46] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225272 (10jcrespo) ``` $ ssh db1059.eqiad.wmnet root@db1059:~$ cat /sys/class/thermal/thermal_zone*/temp 56000 49000 root@db1059:~$ megacli -AdpBbuCmd -GetB... [15:51:19] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225325 (10Marostegui) I would support doing a switchover to db1049 just to be on the safe side for next week switchover [15:53:07] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225328 (10jcrespo) Manuel, I am looking at some options now with Chris (air flow, PSU, ...) , we (and I mean I) will think about that on Tuesday depending on... [16:00:02] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3225331 (10waldyrious) Those weren't really sensitive (in e.g. the legal sense) documents, just internal organizational stuff that didn't make sense to be published, so the... [16:38:41] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225462 (10jcrespo) ``` root@db1063:~$ megacli -AdpBbuCmd -GetBbuStatus -a0 | grep Temperature Temperature: 49 C Temperature :... [16:48:13] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225472 (10jcrespo) 05Open>03Resolved a:03Cmjohnson Executed: ``` megacli -LDSetProp -NoCachedBadBBU -Immediate -Lall -aAll ``` ``` $ megacli -LDInf... [16:53:52] 10DBA, 06DC-Ops, 06Operations: db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) - https://phabricator.wikimedia.org/T164107#3225479 (10Marostegui) Excellent news!! Thanks guys for working this out successfully!! [17:00:01] 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3224848 (10jcrespo) [17:00:24] 10DBA, 07Schema-change: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3224848 (10jcrespo) [17:00:48] 07Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3224848 (10jcrespo) [20:48:48] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3226250 (10Tbayer) [20:49:01] 10DBA, 10Analytics: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#2983518 (10Tbayer) Filed as a new task focusing on the upgrade (since this one had been closed and there was no response here) [21:01:29] 10DBA, 06Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3226286 (10Ottomata) @Cmjohnson Cool, I've announced the date. Let's do it. [21:20:09] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3226303 (10jcrespo) 05Open>03declined MariaDB 10.2 is not stable for production usage as I am writing these lines, it wasn't on T156681, an... [21:22:42] 10DBA, 06Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3226320 (10jcrespo) This wednesday is the failover, do you really want to do it then? We may need Chris or me to put things down and we may be unavailable? [21:28:01] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3226328 (10jcrespo) > That would put us into next FY Q1 for decom of db1047. Let's buy 2 new servers, and reassign them if the functionality is replaced. Knowing typical medi... [21:45:11] 10DBA, 06Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3226386 (10Ottomata) OO interesting. Yeah @cmjohnson that IS a bad time. When else do you wanna? [21:46:03] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3226387 (10Ottomata) Hm, ok, fine with me. Should we not do T159266 then and just wait for new boxes? [21:46:12] 10DBA, 06Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3226388 (10Cmjohnson) @ottomata: is there a better time this week or do you push it out to next week? [21:54:41] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3226400 (10jcrespo) No, let's do that still- assuming no parts will be bought. No harm on a quick reboot and we will not by anything until next fiscal year (months away). [22:06:26] 07Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3226427 (10jcrespo) running: ``` ./software/dbtools/osc_host.sh --host=$host.eqiad.wmnet --dblist=$shard.dblist --table=categorylinks "add p...