Improve Autosuggestions

[angrezichatterbox]

Contributor checklist

• [] This pull request is on a separate branch and not the main branch
• [] I have tested my code with the ./gradlew lintKotlin detekt test command as directed in the testing section of the contributing guide

---

Description

This PR is to address the issues with the autosuggestion regarding the accuracy and the files which weren't required for the project.

Related issue

• Closes Emoji suggestions are not shown for matching keywords #637

[github-actions]

Thank you for the pull request! 💙

The Scribe-Android team will do our best to address your contribution as soon as we can. The following are some important points:

• Those interested in developing their skills and expanding their role in the community should read the mentorship and growth section of the contribution guide
• If you're not already a member of our public Matrix community, please consider joining!
  • We'd suggest that you use the Element client as well as Element X for a mobile app
  • Join the General and Android rooms once you're in
• Also consider attending our bi-weekly Saturday developer syncs!
  • Details are shared in the General room on Matrix each Wednesday before the sync
  • It would be great to meet you 😊

Note

Scribe uses Conventional Comments in reviews to make sure that communication is as clear as possible.

[github-actions]

Maintainer Checklist

The following is a checklist for maintainers to make sure this process goes as well as possible. Feel free to address the points below yourself in further commits if you realize that actions are needed :)

• Tests for changes have been written and the unit test, linting and formatting workflows within the PR checks do not indicate new errors in the files changed

  • Tests may need to be reran as they're at times not deterministic

• The CHANGELOG has been updated with a description of the changes for the upcoming release and the corresponding issue (if necessary)

[angrezichatterbox]

This would require some additional work. Drafting a PR here to have this open to people's suggestions.

[angrezichatterbox]

I have updated the data for the autosuggestion but we would need to attribute them they have a CC license. @andrewtavis would you be fine with it

[angrezichatterbox]

I am not sure how much of an improements it is for english but could you have it checked for german. Also could you check for german and english swear words being censored by the keyboard. I have disabled swear words being generated as suggestion as well. It was present earlier. A small hicup from my end.

[andrewtavis]

Thanks for sending this along, @angrezichatterbox! I'll take a look :)

[Roniscend]

@angrezichatterbox @andrewtavis just a heads up will this pr also include emoji suggestions

[andrewtavis]

Will you be collaborating on this, @Roniscend? We could change this to close #637 then?

[andrewtavis]

Also @angrezichatterbox, us attributing this with CC would be totally fine. Ideally we'd add the attribution to Scribe-i18n and then put it in the third party license sub menu :)

[andrewtavis]

And trying this out now, @angrezichatterbox :) Generally all seems to be working well. There are a few bugs that we can maybe just work through over a call. They're not related to this PR.

note: The filtration of swear words is basically not working. I tried F**k in English and Sch**ß in German and not only could they be completed, but there were autosuggestions 😅

Honestly I'm not really to preoccupied with this, but we do have a working solution. In Scribe-Data we have src/scribe_data/wikidata/query_profanity.sparql, which returns a list of words that are marked as profanity, pejorative (used to disparage someone) or vulgar. We could add an end point for this and populate a table with these values?

We basically would not complete or suggest any string that includes a word in this dataset? This would solve issues like F**ker, which was a suggestion we were getting, etc.

CC @DeleMike and @axif0 👋 How do the two of you feel about adding a profanity table that we then can check against and is updated against the rest of the data?

[Roniscend]

Will you be collaborating on this, @Roniscend? We could change this to close #637 then?

Ok Sure

[andrewtavis]

Updated the PR message to reflect that we'll close #637 with this PR.@Roniscen, let us know anything that needs to change here :)

[angrezichatterbox]

Honestly I'm not really to preoccupied with this, but we do have a working solution. In Scribe-Data we have src/scribe_data/wikidata/query_profanity.sparql, which returns a list of words that are marked as profanity, pejorative (used to disparage someone) or vulgar. We could add an end point for this and populate a table with these values?

We would have to use this. Would this be a lot if not we could load this into a hashmap in the keyboard start or maybe re generate the dicts using the profanity db you would provide.

[angrezichatterbox]

Also the easiest solution for this would be to shift to the Scribe Data one as primary but it is giving me some responses with 2 words which - in betweem like

which -> which-drawing which-kid-is-doing-what

If this gets fixed we would make it like the fallback is the native suggestion module until we generate our own dicts.

[andrewtavis]

notes from the call we're doing, @angrezichatterbox:

which -> which-drawing which-kid-is-doing-what

For this and similar cases we can take the first word to make sure that we are fitting something within the key and are only doing single word suggestions. Specifically also for context, this is important in cases where the user would be confused by a long suggestion that wouldn't fit on the key and might appear to be something else based on how the string is truncated.

We also made scribe-org/Scribe-Data#710 for the profanity data :)