[00:20:15] 10DBA, 06Community-Tech, 07Schema-change: Add local_user_id and global_user_id to localuser table in centralauth database - https://phabricator.wikimedia.org/T141951#2517589 (10kaldari) [00:36:45] 10DBA, 06Community-Tech, 07Schema-change: Add local_user_id and global_user_id to localuser table in centralauth database - https://phabricator.wikimedia.org/T141951#2517641 (10Catrope) During {R325} @Anomie and @tstarling suggested not including the local user ID was a design flaw in CentralAuth that should... [05:17:50] 10DBA, 06Operations, 07Availability: Setup automatic failover for misc database servers - https://phabricator.wikimedia.org/T141547#2517919 (10Krinkle) [06:51:16] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2518050 (10Amire80) [09:01:13] 10DBA, 06Performance-Team, 07Availability, 05MW-1.27-release-notes, and 2 others: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) - https://phabricator.wikimedia.org/T111266#2518232 (10jcrespo) This is now done at database level. One can check: * The lag t... [09:05:27] 10DBA, 06Operations: Display that lag on tendril and dbtree from pt-heartbeat instead of Seconds_Behind_Master - https://phabricator.wikimedia.org/T141968#2518246 (10jcrespo) [09:07:36] 10DBA, 06Operations, 13Patch-For-Review: Puppetize pt-heartbeat on MariaDB10 masters and its corresponding checks on the several monitoring backends - https://phabricator.wikimedia.org/T114752#2518262 (10jcrespo) 05Open>03Resolved Resolving this, T141968 will handle tendril/dbtree and T126757, other moni... [09:08:04] 10DBA, 06Operations: Display that lag on tendril and dbtree from pt-heartbeat instead of Seconds_Behind_Master - https://phabricator.wikimedia.org/T141968#2518246 (10jcrespo) p:05Normal>03Low [09:25:06] 10DBA, 06Operations, 10ops-codfw: BIOS upgrade on certain codfw machines - https://phabricator.wikimedia.org/T139714#2518289 (10jcrespo) 05Open>03Resolved a:03jcrespo Everthings seems ok at the random servers I checked, it sayst bios upgrade, then no strange logs. [10:02:05] 10DBA, 13Patch-For-Review, 05Prometheus-metrics-monitoring: Prepare mysql account and options for prometheus - https://phabricator.wikimedia.org/T128185#2518381 (10jcrespo) 05Open>03Resolved I would say we close this as resolved - we will use the default options for now, which already contains plenty of... [11:31:45] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2518656 (10Nikerabbit) We did not pick up this for our current sprint because: - it seems that the lag... [12:35:40] 10DBA, 10Flow, 03Collab-Team-Q1-July-Sep-2016, 13Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#1828148 (10jcrespo) a:05jcrespo>03None [12:37:30] 10DBA, 06Operations, 07Upstream: TokuDB crashes frequently -consider upgrade it or search for alternative engines with similar features - https://phabricator.wikimedia.org/T109069#2518719 (10jcrespo) 05Open>03Resolved a:03jcrespo The alternative is clear: we are going to convert labs and dbstore (when... [14:23:56] 10DBA: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2518982 (10MarcoAurelio) [14:25:36] actually, that looks like a race condition of the rcstream [14:25:54] where it sends a notification but in reality it is not performed [14:26:21] 10DBA: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2518996 (10MarcoAurelio) In fact, the whole notifytranslators log seems to be broken since June 2016: https://meta.wikimedia.org/w/index.php?title=Special%3ALog&type=notifytran... [14:31:25] 10DBA, 10MediaWiki-Logging, 10Wikimedia-Stream: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519014 (10jcrespo) This looks like a race condition on the rcstream- if an operation fails, it should not be logged and everything... [14:35:43] 10DBA, 10MassMessage, 10MediaWiki-Database, 10MediaWiki-Logging, and 3 others: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519026 (10jcrespo) Sorry for the tag-spam, fell free to revert those. In particular some, such as... [14:38:00] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10MediaWiki-extensions-TranslationNotifications, and 2 others: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519046 (10MarcoAurelio) [14:40:22] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 10Wikimedia-Stream, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519050 (10Nemo_bis) [14:44:08] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 10Wikimedia-Stream, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519082 (10MarcoAurelio) This happened when tried to send a no... [14:57:28] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 10Wikimedia-Stream, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519127 (10Nikerabbit) I also see some sendEmail / Translation... [15:02:55] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 10Wikimedia-Stream, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519156 (10Nikerabbit) >>! In T141988#2519082, @MarcoAurelio w... [15:08:19] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 10Wikimedia-Stream, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519167 (10MarcoAurelio) >>! In T141988#2519156, @Nikerabbit w... [16:13:23] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519415 (10Krenair) [16:36:37] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519550 (10Glaisher) [16:40:44] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519567 (10Glaisher) >>! In T141988#2519156, @Nikerabbit wrote: >>>! In T141988#2519... [16:44:10] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519601 (10Glaisher) Note: The actual issue for the long duration of the submission... [16:53:02] jynus: If an update query is executed within a transaction, will the time taken for it be known only after the commit is issued? [16:53:28] "the time taken for it" is a vague concept [16:53:37] if you update a row within a transaction [16:53:52] it will notmally not be visible until the transaction commits [16:54:05] at least on our default level of isolation [16:54:38] you can know how much time it took individually for the query and the transaction, but those are 2 different measurments [16:54:43] does that help? [16:55:12] normally transactions should be very small in size (think <1 second) [16:55:41] whatever you do inside should fit in that length of time [16:56:04] if you need to do something "transactionally" and it takes more than 1 second, there is a trick [16:56:16] Say, I do this BEGIN; update query 1; update query 2; commit; will the console be immediately available after the update query 1 is issued and will it take the actual time only after the commit is run? [16:56:20] jynus: ^ [16:57:00] I think in the case of mysqli, queries are syncronous [16:57:07] or at leasy by default [16:57:16] so they get blocked until they finish [16:57:29] *least [16:57:45] I'm talking about MediaWiki [16:57:47] trying to understand what's happening in T141988 [16:57:51] so get blocked waiting for query 1, then for 2, the blocked for the commit [16:58:13] T141988: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988 [16:58:14] usually running uncommited queries is very fast compared to the final commit [16:58:46] yes, I assume mediawiki uses default mysqli driver [16:58:58] I'll try to see if I can explain on the bug [16:59:02] but let me check the database code to be 100% sure [17:00:35] to give you an idea: on update, mysql only writes to memory, on commit it writes to disk, so it tends to take more time [17:02:39] jynus: so one part of the code is run with updates, then another part is run with updates without getting blocked and then it only waits the actual time at the end of the request during the final commit [17:02:46] I guess that's what happened in that bug [17:18:35] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-TranslationNotifications, 07Performance: After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia - https://phabricator.wikimedia.org/T141988#2519765 (10Glaisher) OK, after looking at the code, I think this is what happened. M... [17:23:02] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2519772 (10dduvall) We've reached our vcpu and memory quota for deployment-prep so I'm una... [17:30:51] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2519803 (10greg)