[05:36:48] <wikibugs>	 10DBA, 10SRE, 10ops-codfw: db2140 crashed due to HW memory errors - https://phabricator.wikimedia.org/T271084 (10Marostegui) Thanks Moritz. @Papaul let me know if you need something else apart from the idrac logs to provide to Dell in order to get a replacement
[05:58:25] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) @CDanis I have merged the above patch and followed your advice and ran puppet on cumin1001 hosts. I went ahead and tried to edit db2142 to add it to x2 leaving it like this: ` # Editing object codfw...
[06:02:10] <wikibugs>	 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10Marostegui) >>! In T268505#6725693, @gmodena wrote: > @LSobanski would sometime tomorrow after 1300CET work? Otherwise, could you maybe suggest timeslots that would work for you?  Around or a bit after 13:00 CET would work for...
[07:29:37] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui)
[07:31:36] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) s4 done ` # /home/marostegui/section s4 | while read host port; do echo "$host:$port"; mysql.py -h$host:$port commonswiki -e "show create table content_models\G" | grep "model_...
[07:31:55] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui)
[09:50:35] <wikibugs>	 10DBA, 10Growth-Team, 10MediaWiki-Watchlist, 10MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), 10Wikimedia-production-error: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10Addshore...
[10:24:14] <wikibugs>	 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10gmodena) @Marostegui ack. We'll kick in ingestion at around 1300CET - or let you know if the plan changes.
[11:06:21] <wikibugs>	 10DBA, 10Growth-Team, 10MediaWiki-Watchlist, 10MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), 10Wikimedia-production-error: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10Marosteg...
[13:35:40] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) s7 done ` # /home/marostegui/section s7 | while read host port; do echo "$host:$port"; mysql.py -h$host:$port eswiki -e "show create table content_models\G" | grep "model_id" |...
[13:35:51] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui)
[13:37:39] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) s1 done ` # /home/marostegui/section s1 | while read host port; do echo "$host:$port"; mysql.py -h$host:$port enwiki -e "show create table content_models\G" | grep "model_id" |...
[13:39:14] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui)
[13:39:28] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) I have started the schema change on s3 - it will take around 8h to complete
[15:03:12] <wikibugs>	 10DBA: Switchover s4 (commonswiki) from db1081 to db1138 - https://phabricator.wikimedia.org/T271427 (10Marostegui)
[15:03:28] <wikibugs>	 10DBA: Switchover s4 (commonswiki) from db1081 to db1138 - https://phabricator.wikimedia.org/T271427 (10Marostegui) p:05Triage→03Medium a:03Marostegui
[15:03:54] <wikibugs>	 10DBA: Switchover s4 (commonswiki) from db1081 to db1138 - https://phabricator.wikimedia.org/T271427 (10Marostegui)
[15:04:24] <wikibugs>	 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui)
[15:04:26] <wikibugs>	 10DBA: Switchover s4 (commonswiki) from db1081 to db1138 - https://phabricator.wikimedia.org/T271427 (10Marostegui)
[15:04:28] <wikibugs>	 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui)
[16:59:16] <wikibugs>	 10DBA, 10SRE, 10ops-codfw: db2140 crashed due to HW memory errors - https://phabricator.wikimedia.org/T271084 (10Papaul)  Create Dispatch: Success You have successfully submitted request SR1048216249.
[18:19:58] <wikibugs>	 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10gmodena) The process finished at around 1900CET. From our end, contention and read/writes stats (grafana) seemed ok throughout the process. Some throughput stats: ` Loading /home/isaacj/sockpuppet/data/2020-12/temporal.tsv: 16...
[23:15:49] <wikibugs>	 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10Etonkovidova) >>! In T267216#6728387, @Marostegui wrote: > We just had another spike of queries like the following on `...
[23:55:30] <wikibugs>	 10DBA, 10Growth-Team, 10MediaWiki-Watchlist, 10MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), 10Wikimedia-production-error: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10MusikAni...