[00:37:24] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991883 (10kaldari) @jcrespo: Everything looks correct on all the servers I was able to connect to.... [01:00:16] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2991922 (10mmodell) incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170201-Phabricator [07:02:45] 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#2992312 (10Marostegui) dbstore1002: ``` root@neodymium:/home/marostegui/git# mysql --skip-ssl -hdbstore1002.eqiad.wmnet metawiki -e "show create table pagelinks\G" ***************************... [07:07:16] 10DBA, 10Analytics: Drop m3 from dbstore servers - https://phabricator.wikimedia.org/T156758#2992314 (10Marostegui) m3 has been removed from dbstore2001 as per: T156905#2991826 [07:31:03] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2992323 (10Marostegui) dbstore1001 didn't have the alter done for enwikivoyage and enwiki. I have a... [07:36:58] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2992331 (10Marostegui) In addition to what @jcrespo mentioned, in general, we are not completely happy if we decommission servers before running pt-table-checksum (T154485) wh... [08:50:42] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2992453 (10jcrespo) a:05kaldari>03jcrespo @Marostegui No server has the change for enwiki and en... [08:52:32] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2992455 (10Marostegui) >>! In T156305#2992453, @jcrespo wrote: > @Marostegui No server has the chang... [09:24:11] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2992507 (10jcrespo) This should be done now, please recheck all servers you can (forget about db1069... [10:17:30] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2992609 (10Marostegui) Hi, Some tests I have been doing: - Executed the same query on db1048 (slave that crashed - OOM). Query took 2 minutes (as it did yester... [13:28:27] marostegui: not sure if temporary but graphite.w.o returns a weird python stack trace ending with "OperationalError: database is locked" [13:28:53] ah now recovered [13:29:19] sort of, the UI kinda loads but it is very slow [13:38:13] elukey, not sure if you know, but graphite doesn't use mysql or postgres [13:38:28] it uses its own storage [13:38:37] look for heavy hitters [13:39:39] I thought that it was using it to store the metrics names etc.. [13:39:41] my bad [13:40:55] jynus: any suggestion about checking for heavy hitter? Never done anything on graphite1001 [13:41:03] (not sure if you have done before) [13:41:09] not really [13:41:21] but I would go to graphite and see what is happening [13:41:34] do ps and that [15:15:16] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#2993436 (10Marostegui) Thanks @Catrope - I will probably not drop any tables on this run, just to be safe :... [15:16:52] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#2993439 (10Marostegui) a:03Marostegui [17:21:01] 10DBA, 10Analytics: Drop m3 from dbstore servers - https://phabricator.wikimedia.org/T156758#2985275 (10Milimetric) We don't know what m3 is. So far we don't know of a use for Phabricator databases on the analytics slaves. [17:21:24] 10DBA: Drop m3 from dbstore servers - https://phabricator.wikimedia.org/T156758#2993958 (10Milimetric) [17:24:07] 10DBA, 10Analytics: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#2983518 (10Milimetric) It was the first time we tried to use it, it would be very useful if we could get it to work though. [18:41:00] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2994282 (10kaldari) 05Open>03Resolved Checked on all database servers for enwiki (s1) and enwiki... [22:40:51] 10DBA, 10Social-Tools, 10SocialProfile: Running update.php with an SQLite-runned wiki while SocialProfile is installed will return a database query error - https://phabricator.wikimedia.org/T157082#2994951 (10SamanthaNguyen)