[01:28:58] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3226802 (10Tbayer) @jcrespo I totally hear you on 10.2 not being ready. But there has to be some way to record this analytics need on Phabricat... [03:15:58] 10DBA, 06Operations, 10ops-eqiad: db1047 BBU RAID issues (was: Investigate db1047 replication lag) - https://phabricator.wikimedia.org/T159266#3226868 (10Ottomata) As long as we give a couple of days heads up, I think we're fine. Pick a day andy day :) Just let me know with enough time to get an email out. [05:24:57] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3226250 (10Marostegui) >>! In T164224#3226802, @Tbayer wrote: > @jcrespo I totally hear you on 10.2 not being ready. But there has to be some w... [05:42:11] 10DBA, 10Analytics: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#3227009 (10Marostegui) Sorry I missed T156681#3195441 but as I said in the previous comment, we were (and are) only evaluating 10.1 for production nowadays. 10.2 doesn't even have a GA yet. [08:08:54] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227167 (10jcrespo) @Tbayer Let me show you how to use a programming language to obtain the desired results: in PHP: ```lang=php $query = "SE... [09:17:10] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3227329 (10jcrespo) A primary key is being created on templatelinks and pagelinks (T164185)- once it is done (and merged the same change on mediawiki) we can con... [10:24:50] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227445 (10Tbayer) >>! In T164224#3227167, @jcrespo wrote: > @Tbayer Let me show you how to use a programming language to obtain the desired re... [10:46:53] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227476 (10jcrespo) > This task is about the ability to do this in SQL and get direct results without adding steps That was declined at T16422... [11:10:27] 10DBA, 06Analytics-Kanban, 06Operations, 15User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3227526 (10elukey) [11:41:53] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227550 (10Marostegui) >>! In T164224#3227445, @Tbayer wrote: > > This task is about the ability to do this in SQL and get direct results wit... [12:38:40] 10DBA, 06Analytics-Kanban, 06Operations, 15User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3227650 (10elukey) About those 66GB: Piwik uses ~18.5 GB of data, but /var/lib/mysql/ibdata1 is 66GB (no innodb_file_per_table set). [12:45:20] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227655 (10faidon) First off, let's tone this down a little bit and try to unpack the conversation to all of the separate contentious points :)... [13:30:42] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227779 (10Ottomata) > it is a bit surprising that this is now a drawback for the solution that was implemented on: T153207 We didn't lose any... [13:47:15] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227811 (10faidon) >>! In T164224#3227779, @Ottomata wrote: >> and hoping to "[get] people of of EventLogging MySQL in Q1 next FY". > I now thi... [13:50:34] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3227830 (10Ottomata) The way we are talking about it, it is the former. We're sure we want T162610, so we are doing it before we think about h... [14:22:49] jynus: checking masters lists: [14:22:57] codfw: db[2016-2019,2023,2028-2029,2033].codfw.wmnet,es[2016,2018].codfw.wmnet [14:23:07] eqiad: db[1031,1052,1054,1061-1063,1068,1075].eqiad.wmnet,es[1011,1014].eqiad.wmnet [14:24:32] they seems the correct one compared to tendril tree to me. [14:25:21] yes, they are [14:25:31] I do not have to look at tendril, I know them by heart [14:25:47] eheheh [14:25:53] thanks for checking [14:26:00] no [14:26:08] thanks *you* for checking [14:50:47] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228005 (10zhuyifei1999) [14:53:15] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228024 (10MarcoAurelio) @zhuyifei1999 Sorry but I'd like to keep my project closed to my eyes only and the admins if they ever need to see what's going on there. However you can see in the description which... [15:14:26] 10DBA, 06Operations, 10netops, 10ops-codfw: db20[7-9][0-9] switch ports configuration - https://phabricator.wikimedia.org/T162944#3228091 (10Papaul) @Robh any update on this? [15:17:05] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228106 (10zhuyifei1999) The necessary questions (which I could have investigate myself if the dir were at least 004) before knowing what went wrong with jstart: 1. do you have a custom $PATH, $PYTHONPATH,... [15:29:00] 10DBA, 06Operations, 10netops, 10ops-codfw: db20[7-9][0-9] switch ports configuration - https://phabricator.wikimedia.org/T162944#3228127 (10RobH) Nope, I forgot about it! I'll knock them out now. [15:30:06] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228129 (10jcrespo) You can see here a summary of the lag of s7, which is where eswiki is hosted (it is fully public): https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?panelId=7&fullscreen&or... [15:43:22] 10DBA, 06Operations, 10netops, 10ops-codfw: db20[7-9][0-9] switch ports configuration - https://phabricator.wikimedia.org/T162944#3228138 (10RobH) row c done [15:59:06] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228192 (10MarcoAurelio) The task is already finished with the template replaced so I'm done here. Thank you to everyone who have helped. Maybe T154011 is the solution for this, I don't know. However and in... [16:01:02] 10DBA, 10Analytics, 10Analytics-EventLogging: Update MariaDB on analytics-store to a version that supports JSON functions - https://phabricator.wikimedia.org/T164224#3228211 (10Nuria) >We didn't lose any functionality by converting this field to JSON. Previously it was a free form user agent string, difficul... [16:04:41] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3228214 (10RobH) [16:04:44] 10DBA, 06Operations, 10netops, 10ops-codfw: db20[7-9][0-9] switch ports configuration - https://phabricator.wikimedia.org/T162944#3228212 (10RobH) 05Open>03Resolved row d done [16:08:30] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3228222 (10waldyrious) On a slight tangent: during the import, we're taking care to avoid importing all pages indiscriminately, to reduce some of the cruft (templates, redi... [16:13:28] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228236 (10zhuyifei1999) https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Using_the_shared_Pywikibot_files_.28recommended_setup.29 is a $PYTHONPATH based setup. AFAIK the griddoes not import env... [16:13:46] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3228238 (10Dereckson) No, we can't. But you can arrange with someone with Terbium access a maintenance window to run the scripts together. [16:23:02] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3228264 (10Dzahn) Which scripts exactly do you want to run? Are we talking about existing scripts on the mediawiki maintenance hosts (terbium/wasat)? [16:29:01] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3228306 (10Dereckson) @Dzahn According https://www.mediawiki.org/wiki/Extension:Maintenance#List_of_supported_maintenance_scripts and the section afterwards, this is a wrap... [17:12:55] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3228499 (10zhuyifei1999) @MarcoAurelio while T164277 is not fixed, a workaround is to set the environment within the script you are running (in this case `/mnt/nfs/labstore-secondary-tools-project/mabot/.pyw... [18:44:27] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3228800 (10Papaul) db2084 can not boot to PXE. I am troubleshooting it. [19:37:27] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3229098 (10Alchimista) The scripts are the ones who feed the Special:SpecialPages Maintenance reports: Dead-end pages, Orphaned pages, Uncategorized[categories | pages | fi... [19:52:02] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3229132 (10jcrespo) Alchimista- unless there is a configuration error, those are setup automatically for you when are part of our production cluster. Wait 1 or 2 weeks, and... [22:30:00] 07Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s3-eqiad - https://phabricator.wikimedia.org/T163912#3229654 (10jcrespo) 05Open>03Resolved This is now ran on all eqiad servers through replication, only codfw is pending (aside from ignored tables). Given the... [22:30:02] 10DBA, 07Epic, 13Patch-For-Review, 05codfw-rollout: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) - https://phabricator.wikimedia.org/T155099#3229656 (10jcrespo) [22:30:05] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3229657 (10jcrespo) [22:58:55] 07Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3229873 (10jcrespo) Applied on all masters: db1052, db1054, db1068, db1063 and db1062. All eqiad slaves pending. [23:21:14] 10DBA, 06Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3229989 (10jcrespo) I am testing mydumper (and myloader) on db1015. That is one of our slowest hosts, although s3 should be favorable for an efficient dump. Taking a full s3 ba...