[08:46:02] JD|cloud: around [15:29:55] PROBLEM - wikidata-alerts grafana alert on icinga1001 is CRITICAL: CRITICAL: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is alerting: Max Lag above 10 for 1 hour. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [15:44:43] RECOVERY - wikidata-alerts grafana alert on icinga1001 is OK: OK: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is not alerting. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [20:21:59] Lydia_WMDE: could someone look into https://phabricator.wikimedia.org/T258150? [20:22:21] as far as I can tell, it's impossible to add the statements via the website, I'm having to resort to doing single edits using quickstatements :/ [20:23:20] @nikki does it just take a while and then it works? [20:23:33] if so then there is another ticket that the search team was looking into [20:23:35] let me find it [20:23:52] I think so, but I don't know how long [20:24:08] https://phabricator.wikimedia.org/T240328 [20:24:34] Erik commented on that one a few days ago [20:25:30] Is that the same thing? then i'll merge and bump it [20:26:25] I'm not sure it is, someone says it's also the case for items, but I'm not seeing it with items [20:26:53] For Items we specifically added a hack to index them faster [20:26:59] so we'd need the same thing for lexemes [20:27:12] the hack isn't working 100% of the time but most of the time [20:29:23] hm, then maybe it is? I don't really know enough about how it works behind the scenes, I just know that I have no way to select the lexeme I want and that's frustrating :) [20:29:45] understandably [20:29:57] so the short version of how i understand it works: [20:30:18] the entity selector uses elastic search in the background to find suggestions [20:30:35] elastic search is updated asynchronously whenever an edit is made [20:31:40] that's usually ok except in the case when you need to create a new item for a statement [20:32:10] at some point we started prioritizing updates that were new item creations to work around that. basically when a new item is created it gets indexed faster than other edits in elastic [20:32:22] and now we basically have the same issue for lexemes [20:33:14] * nikki nods [20:35:06] i'll merge the tickets and bump it [20:35:13] it has to be handled by the search team though [20:35:15] ok, thanks [20:35:42] no problem [21:37:59] PROBLEM - wikidata-alerts grafana alert on icinga1001 is CRITICAL: CRITICAL: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is alerting: Max Lag above 10 for 1 hour. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [22:18:43] RECOVERY - wikidata-alerts grafana alert on icinga1001 is OK: OK: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is not alerting. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [23:29:15] PROBLEM - wikidata-alerts grafana alert on icinga1001 is CRITICAL: CRITICAL: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is alerting: Max Lag above 10 for 1 hour. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [23:55:13] RECOVERY - wikidata-alerts grafana alert on icinga1001 is OK: OK: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is not alerting. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/