[05:29:42] wow 26 secs lag on test.wikidata what's going on there? [10:33:03] PROBLEM - High lag on wdqs1003 is CRITICAL: 4383 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [10:59:39] RECOVERY - High lag on wdqs1003 is OK: (C)3600 ge (W)1200 ge 1189 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [14:01:16] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @cscott - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:51:05] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @cscott - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:11:44] PROBLEM - High lag on wdqs1010 is CRITICAL: 5631 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:12:06] PROBLEM - High lag on wdqs2005 is CRITICAL: 5631 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:12:34] ACKNOWLEDGEMENT - High lag on wdqs1010 is CRITICAL: 5631 ge 3600 Gehel catching up on updates after data reset https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:13:04] ACKNOWLEDGEMENT - High lag on wdqs1010 is CRITICAL: 5631 ge 3600 Gehel catching up on updates after data reset https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:13:04] ACKNOWLEDGEMENT - High lag on wdqs2005 is CRITICAL: 5631 ge 3600 Gehel catching up on updates after data reset https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:44:54] RECOVERY - High lag on wdqs1010 is OK: (C)3600 ge (W)1200 ge 1145 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:54:39] RECOVERY - High lag on wdqs2005 is OK: (C)3600 ge (W)1200 ge 790.3 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [17:06:33] is it me or no Wikidata JSON dump was generated this week? [18:43:27] Envlh: Yeah, we're having troubles… but will probably have a new dump by the end of the week [21:56:55] hoo: aww, just saw that :( thank you for your work!