As I continue to use Frankie in more ways and with different types of content, I’m beginning to see some friction points. Some are just a function of the limited graphics of the TUI interface I’m using, but some are about the actual information layout itself. Before I start tinkering with it, I want to talk my way through it. And that means: It’s rubber duckie time again!
FrankenTongues began with some very specific features in mind - features that I couldn’t find in other tools - and it’s taken a while to get here, but today was the day I finally got to play with two of the biggest.
Most language tools I’ve encountered focus on word-level learning - studying individual words and their meanings - but Frankie takes a higher level view, letting me study full sentences. I’ve been using it to read books one sentence at a time for a while now, but my architecture was designed to let me do much more than that, and I’m finally getting to see if it will be as useful as I’ve been hoping.
The Earworks Lab
The first requirement is fairly straightforward: In order to add Frankie’s ear-training features, I need to be able to create audio-based lesson content and then be able to play it back. The rudiments of audio storage and playback were worked out when Frankie was just a command-line tool. I talked about integrating that into the CLI the other day, but that’s as far as I got. The TUI interface still wasn’t even able to play simple audio. And since that’s the version of Frankie that I’m actually using every day, it meant my ear training has been going untrained all this time.
Until now.
Today I actually loaded one of the cut up podcast lessons into the TUI and played it through, listening to each Norwegian sentence, replaying it at will, listening to the English audio, opening the English or Norwegian text… I can even play the slow Norwegian audio when I have trouble parsing the audio at normal “conversational” speed.
What this all boils down to is that, after a several months of work, I’m finally able to use the feature that inspired me to build this thing in the first place, and I couldn’t be happier with the result. It feels exactly as effective as I’d imagined.
Dynamic Lessons
The second achievement came as almost an accident. I don’t think I’ve talked about this detail before, but Frankie is built on an interesting database concept. The key record in the database is the sentence, but it’s neither a text sentence, nor audio. Instead, I think of it as the platonic ideal of the sentence. A lesson document is made up of sentences in a particular sequence. If I load Moby Dick as the first book in the app, it will be book 1, and its first sentence can then be referenced as just (1,1). First book, first sentence. The second sentence will then be (1,2), the third will be (1,3) and so on.
The real fun lies in the table of renderings. The familiar text “Call me Ishmael.” is stored as doc=1, sentence=1, language=English, medium=text, but that’s just one rendering. I also have “Kall meg Ismael.” saved as the Norwegian text entry. Then there’s a recording of me reading the sentence in English, with medium=audio, and a slowly enunciated AI reading with medium=slowaudio.
In this system, every sentence of every document is rendered multiple times, and can then be used in multiple ways, depending on the purpose of the lesson. My data system so far comprises 5 podcasts and 5 novels. Each of them is fully rendered in both English and Norwegian, in each of text, audio, and slow audio forms. And today I unlocked the first attempt to make use of all that in the feature for which the app itself is named.
A FrankenTongues lesson can gather as many body parts as it wants, from anywhere - any sentences from any documents with any features. Do you want to study nuances of the verb “vurdere”? You can build a lesson of only sentences that include “vurder*“in the Norwegian text. Do you want to review all the text sentences that you’ve studied at least 10 times but still have a reading competence score of less than 50%? Or maybe you want to take all the sentences longer than 15 words that you’re solid on for reading, and see how well you can follow them in audio.
In theory, anything that can be expressed as a SQL search can be the basis of a lesson, because under the hood, that’s how all lessons are built. Even a simple one that lets me just read a book is built by querying the DB for every sentence with that document id, sorted by sentence sequence number. The reader screen is just stepping through the sentences returned by a SQL query, so it doesn’t care how the list was built. It can work with any query.
Previously, when I’ve been listening to a book in audio sentence mode, I’ve felt that I was getting too much advantage from the context. I knew who was in the room and what they were doing and what they were talking about and what the mood was, so it was easy to pick out the words. My ear was already primed with all the most likely words and concepts. That made following the story easier, but my ear was not having to work as hard as it should. After all, this is supposed to be a workout, right? Shouldn’t it be hard?
Well, in today’s exciting step, for the first time, I was able to test that question by creating a FrankenTongues lesson drawn from the entire database. I took all the sentences, from all the books and podcasts, and sorted them by length. This effectively ripped them out of their contexts and forced me to figure out everything from the audio itself. No context priming possible.
And it was fun! I’m definitely going to enjoy building these custom lessons as a way to drill down on things. And as an added bonus, any time I add a new document to the system, all the previous lessons potentially get new content to work with, because now their defining SQL searches will pull in more sentences.
This first proof of concept test was based on a hand-coded SQL query, because I haven’t got an interface set up yet for describing new search criteria, but I’m just dancing with delight at how it’s all coming together.
And I’m confident now that as soon as I get this working on Android, I’ll be able to really start slamming the ear training.
