My learnings from bot devs and first-hand localizer’s advice (part 1: workflows, i18n & l10n)

This post was originally published in Chatbots Magazine.

The interactions of the future are not made of buttons. 

— UXdesign.cc

Anyone with or without dev skills can build a simple conversational bot — what Phil Libin of General Catalyst calls a “fart bot.”

But how about building quality polyglot bots?

Even most of the top bots on Product Hunt and elsewhere speak only one language, most often this is English. Multilingual bots are few and far between, and this is for a good reason.

This is partly because bots are in their early days, but it’s also because rolling out a quality social bot able to speak foreign languages and understand them (at least to some degree) is an ambitious project. It takes longer than a weekend to pull off, and is yet to be fully attainable today — the technology needs to take that leap.

In our chat on Facebook, Adit Jain, an indie bot developer, said:

Non AI-based bots with carousels, buttons can be created in other languages with ease as they are just like apps. In the case of AI-based bots, it is a completely different ball game. The amount of work you have to put in is at least 100 times of that. [And for the bot to understand a foreign language] the developer or someone he/she is working with must understand the language for this. 

— Adit Jain, a bot dev

It’s a challenge, but an opportunity too: The more languages you can teach your bot, the more clients will be interested in buying your services as a bot dev.

Adding languages to a bot is a UX problem and not simply the matter of translating all text. Think of it as a redesign project or building new versions of your bot and this approach should help ensure a bug-free, enjoyable UX.

This is a 2-part article. In this part, I’ll share some workflow insights I’ve collected from bot devs and also my localizer’s perspective on building with future localization in mind, shaping a localizable personality, crafting a localizable script, and even the dangers of your bot using certain emojis🙌 when socializing with foreigners.

A short disclaimer: I’m not affiliated with any of the below startups or developers, nor am I aiming to promote them.

# WORKFLOWS THAT WORKED FOR OTHER BOT DEVS

The chatbot ecosystem is fragmented.

As a dev, you know how many tools, languages, libraries, and platforms there are already. Which ones do you choose?

Below are some workflow ideas I’ve collected from bot developers.

ChatShopper’s Way

In our conversation on Facebook, Matthias Nannt, the founder of chatShopper, shared with me how they build their multilingual bot:

At first we only had the German version of our bot. We are using wit.ai as an NLP system and a Node.js backend for the bot logic. As we started adding multilanguage capabilities, we first replaced the bot answers we stored mainly in wit.ai’s bot-engine with placeholders we resolve in our bot logic based on the language. Then we trained a new wit.ai instance to handle conversations in English (clone of the existing German wit instance with new English training data). For the answers of our bot we also copied the German language file, translated it ourselves, and let a native speaker do some corrections.

We use the locale set by the user in his/her FB profile to choose which languages the user gets and added a language change option to the bot. From that point on, with every change to the bot we needed to make sure, that both wit.ai instances have the same functionality what makes developing and deploying new features more complex. Translating output of an app and a chatbot is nearly the same. You have placeholders in the frontend and language files in the backend that contain the strings used in the app/bot based on the user’s language. The main difference in a conversational interface like we have with our chatbot is that you also have a complex language input by the user that needs to be handled by an NLP engine. So, translating a chatbot is not just translating the output message, but also training an NLP model for the new language.

Developing bots at the moment is a bit more unstructured and you don’t have a fully functional development environment like you have for apps (Android Studio or xCode). You just get messages from Facebook (or other messenger), do something, and send a response back to the user. What you do in the “do something” part is totally up to you. You can choose whatever programming language you like and design it in the way you like. So there are different ways to implement multi-language functionality. But there are existing i18n patterns and plugins for different programming languages that all mainly function the same… It’s kind of best practice. 

— Matthias Nannt, chatShopper

Of course, not all bots will need AI capabilities.

If you opt for NLP, it’s worth remembering that while it may work for English, NLP for other languages doesn’t cope as well.

Visabot’s Way

Here’s how Visabot’s team implemented multilingual capabilities for their bot:

We decided to build a two-language version from the very start and do it from scratch. This approach removes any limitations and constraints which we would inevitably run into had we chosen to go with a bot builder. Speaking of multiple languages, it was relatively easy: During the first interaction, we ask the user their preferable language and show options to choose from. The bot uses that language. The script is stored in a database, and similar to how it’s done for multilingual sites, to ensure the right database architecture, we implemented language selection in the least complicated way. We used a pretty simple approach for building multilingual functionality. But, in principle, there are no limits for creativity here, as the developer works on the server side. It’d be very difficult to achieve the same, if we worked in the builder. Builders are good for those who doesn’t want to code.

The trickiest thing is when a user sends an emoticon or suddenly starts writing in English. 😊 

— Artem Goldman, Visabot

Visabot’s team plans on adding support for Spanish and Chinese next year.

For the cases when your user switches to another language you may use a language detector in your conversations that will recognize a different language based on the percentage of words in an utterance. Then your bot can offer the user to switch to the language with the highest proportion.

One Indie Dev’s Insights

I reached out to Vic Yankoff, who had build polyglot bots before, to learn how he would implement multilinguality. Here’s the gist from our Slack chat:

A great way to localize your chatbot is using Google Translate’s API. Google Translate is not very good about giving accurate grammatical translations, but is excellent about translating keywords which is often good enough for chatbots to understand what the user is asking or saying. I know many people would disagree, but I also like using buttons in my chatbots because they are really easy to localize. With the buttons, the developer knows exactly the meaning and action to take, so the only difficulty is translating the button labels since the postbacks are controlled and known in advance by the developer. Not sure about the other bot platforms, but Facebook makes localization really easy by giving you the language the user is using on Facebook. This is abbreviation like en (English), de (German), sp (Spanish) etc… Each time a new user tries the chatbot, the developer can detect what Facebook language preference they have, and invoke the correct language in the chatbot. Debugging a multilingual bot is entirely different can of worms and something that needs to be done more carefully because you need someone that understands what the user is saying.

The benefits of multilingual bots are enormous. They are hard to get right, but if you succeed, the benefits and number of new users and markets you tap into are astronomical! 

— Vic Yankoff, a bot dev

If you need help adding languages to your bot, Vic will be glad to chat with you. Feel free to reach out to him via his LinkedIn profile.

Make Your Resources Easily Accessible

Plan where is best to store your translation files. Maybe it’s best not to store them in your code repository: your linguists should have access to files early.

If you plan on training the NLP engine, linguists will need a way to come back and make adjustments. Make it easy for editors/linguists to get at the input/output messages, so you can farm off the language editing component to them.

Suggestion: Google spreadsheets integrate well with machine translation — using MT, you can get a fast feel of what is being discussed in the languages you don’t know and the training the linguists are providing.

Here are more insights on the whole process and i18n from Klemens Zleptnig (Toni bot).

When is the Right Time to Start Adding Languages?

Before adding languages ask yourself if your users actually want to talk to your bot in their native languages. For most cases the answer would be yes. It’s a well-known fact that we like using products in our native languages.

To decide which languages to add first, analyze what countries your current users are coming from and what their preferred languages are. Also, consider how strong your competition is in those countries and what your stakes are.

A great idea or product is awesome. But only when it’s timely, might it be worth millions

If you think your bot has big potential, it makes sense to think big, plan ahead, and roll it out in other languages to capture the foreign markets early — before others do this.

Microsoft released WeChat bots in China and Japan (Xiaobing, Xiaoice, Rinna, and Toy) — obviously, Microsoft is poised to bite a piece off the Asia’s bot pie, fast.

Fast, though, doesn’t mean rushing things: Teach your bot one language first. You’re luckier, if you’re a native or proficient English speaker, for obvious reasons.

From my other Facebook conversation:

If there are budgetary constraints and English is your native language, stick to one language version: focus on getting the English chatbot “perfect,” learning what works and what doesn’t, getting to product/market fit for a problem that truly needed solving PRIOR to exporting to other countries and languages. 

— Sat Panesar, a bot writer and dev

This strategy aligns well with a lean startup approach.

In one other chat, Matthias Nannt shared:

Get as much learning as possible in one language and iterate fast. When you are sure what to build and how (and things don’t change that often), build it with i18n in mind so you can always add second (and later third…) language when it makes sense. 

— Matthias Nannt, chatShopper

Learnings: Plan ahead. Start with one-language solution, code it to work with different languages from the start but don’t implement them yet; polish it, constantly tweak the script to improve the flow and responses based on what you’re getting from users; once the concept has been proved, start adding more languages.

# BUILD FOR FUTURE LOCALIZATION

It’d be only fitting here to reemphasize the importance of i18n. Especially to young developers with no prior development experience trying their hand in bot development.

To save yourself the trouble of reworking your code later on, make sure to review established i10n and l10n best practices and transfer them to the area of bot development.

There is a ton of resources online. See, for example, this post written by my fellow team member, or see a small list on the very bottom.

Most of the advice will be, if not directly applicable, very helpful when building, especially when building from scratch.

Before releasing, test your bot’s UI against l10n and i18n bugs. Implementing i18n and testing should help you avoid some common localizability (l12y) bugs — such issues as croppings/overlaps, wrong date/currency formats, missing characters.

Build “foreign first:” It’s very difficult to retrofit me once I have been created

# CRAFTING A LOCALIZABLE SCRIPT

A lot of users responded [to the bot] as they would to a human, and when they got non-human responses, they’d stop using it. 

— The Guardian team

It’d be wrong to assume your bot’s responses will work in other languages after localization just because they do in the main language. Below are just a few examples of areas where things can go wrong and suggestions for improvement.

Variables and Different Word Forms

You might want to take the user’s input and use it in your bot responses at runtime; however, taking a word or phrase out of context and putting it into another sentence works fine in English, but breaks in other common languages. This will give away the feeling of speaking to a dumb machine when your bot’s goal is to speak the least robot-like.

Many languages have inflections and cases, so where in English you have one form of a word, in other languages there can be a variation (form) of this word and in another context yet another variation.

Here’s an example of using user’s intent as a variable going south in localization (and I’ve seen a very similar one actually being used in one bot builder docs):

The same exchange in Russian, simply using the user’s input and not accounting for the case change, results in a wrong grammar use:

How to improve: In your code, instead of placeholders/variables (unless the bot has been coded to handle this properly) use full sentences and send those to localizers. Also don’t reuse a string in different contexts as it might get an opposite meaning.

If using placeholders (variables), add comments for your localizers explaining what each placeholder will be showing:

You may also use graphic elements instead.

For example, when giving a selection of pizza toppings, you might just need a few images or emojis. Besides, this way, the user will need to type less.

Some icons might have a different meaning in other countries or foreign users simply might not recognize an icon because in their country designers use a different one.

A good localizer will advise you if there’s any problem with any icons or emojis in their culture or simply will localize it as any other content — replace it with a more suitable one.

Another example is from Roll bot (a group chat bot from Kik devs), that uses variables to make it more human-like and personal. When a user is being offensive to Roll, the bot reacts “with a response that substitutes in the name of the user who produced the offending content.”

While bot’s responses work fine in English, they wouldn’t work that nice in inflected languages (all Slavic languages are, Japanese too) like Russian and Ukrainian (native for me). When the bot simply reuses the user’s intent, unless it has been coded to handle this, the user’s name will be in a wrong case (grammatical form) in these bot’s responses:

I’ll be taking questions from <name> less serious from now on…

Can someone besides <name> ask me something?

This is just an example for illustration purposes, Roll hasn’t been localized into other languages.

How to improve: Don’t rely heavily on variables. Think about showing visuals instead of text. But remember, some images, icons, and emojis might not be suitable in certain countries — more on this below.

Expansion

While it’s very tempting to use carousels, buttons, and quick responses (they ensure user input is consistent and you save the user from typing etc., thus reducing friction), but it’s a room for UI bugs introduced during localization if you don’t plan ahead.

For example, Messenger allows only up to 80 characters in title and subtitles, everything that exceeds that will have to be reworked. Remember to point this out to localizers (e.g. add localization comments), otherwise, you’ll have to get back to them and ask for shorter versions (from experience, this happens quite often).

Here are Messenger’s detailed guidelines on sizes for titles and subtitles (and image dimensions).

Other places to watch for croppings: calls to action, carousels, conversable forms (HTML-like elements such as checkboxes, checkmarks, and radio buttons in chat bubbles), and in persistent menu buttons.

Croppings or overlaps are common UI bugs. A developer from Mozilla actually statistically calculated how much text expands after translation.

A chart to the right shows proportions for different languages (English is 0 and those are language ISO codes).

The chart to the left illustrates that only a handful of languages are more concise than English according to this dev’s effort. Here’s the original post with findings.

Devs can use this chart to estimate text expansion.

If you have the UI mockups with all the text in place, you’d make localizers happy if you sent those to them.

This way they could not only see the text in context, but could even translate in context by opening them in an editor and seeing how much room they had.

One more tip: John Saito suggests using Google Spreadsheets in combination with the GoogleTranslate function to approximate translation lengths: You can quickly machine translate your strings in multiple languages to get the feel of how much leeway you need to have.

How to improve: Follow best practices, avoid verbosity, be concise when writing for your bot. In bot responses, you may want to use non-text elements like images, video (Messenger and Telegram), audio (Messenger, Telegram, and Sequel), and icons/emojis instead of text. If using buttons or quick responses, account for text expansion and make sure to provide localizers with character limits they should stick to.

Give Your Bot a Gender

If your bot is a native English speaker, it’s understandable if you didn’t even think about what gender your bot is. But when translating into certain languages, the gender question will be hard to avoid.

Some parts of your script might be untranslatable unless your translators know whether your bot is female or male. In many languages pronouns, verbs, nouns etc. have different forms depending on the gender (yep, English is that gender-friendly).

In something simple like “I’ve told you,” the gender-neutral ‘told’ in other languages will be translated differently depending on whether your bot is female or male

Translators might be able to find a creative walk-around solution, but not all the time.

One example of a gendered bot: Microsoft made its Toy bot to be a 19-year old American girl.

The fate of Toy bot is also an example that slipping in delicate matters, be it gender or politics, can have most unpleasant consequences (it was phased out shortly after the release).

How to improve: Be sure to plan ahead prior to writing the original script and tell your translators upfront whether your bot is female or male. Giving it the neuter gender will most likely raise eyebrows (it does in Russian and Ukrainian).

Avoid Using Machine Translation

And if you decided to use Machine Translation, be aware that your bot might become sexists for the same grammatical reasons as above.

When translating from gender-neutral to gender-specific languages (and vice versa), statistical MT will inevitably have to attribute this or that gender, thus it’ll be essentially playing statistical guessing games (leaning towards masculine). And because AI has trouble getting the context, this issue is hardly programmable.

How to improve: Even though Neural MT is making strides, for now, rely your business on human translators. If you’re on budget, consider using a hybrid MT+escalated human in-chat translation; there are such solutions on the market already.

Emojis Can Get Dangerous 😈

A UK-based language company recently ran a job advert — Emoji localizer needed. The responsibilities would involve “explaining cross-cultural misunderstandings in the use of the mini pictures.”

Emojis help eliminate ambiguity and add context and emotional layer to the conversation. If mishandled, though, they can spark an international brawl… or just a frown.

SwiftKey did an extensive research on emojis use in various countries. Here are some of the findings of interest in this delicate area (a professional localizer will be sure to use these in their work):

• Substitute smiley faces with hearts for French users: 55% of emojis sent in French are hearts vs. 8% in English and US Spanish, where smiley faces prevail
• LGBT emojis are OK in the USA, but you’ll want to forgo them for some other countries
• Alcohol-themed emojis are high in Australia and the US, but limit their use when localizing for Turkey and Arabic countries.

To avoid any cultural inappropriateness, consult your linguists, as there are gestures that might be misunderstood abroad.

You might also want to instruct your linguists to consider even the color of human-themed emojis.

This must be an important issue if Apple has introduced racially diverse emojis in iOS and Slack introduced a welcoming hand that doesn’t discriminate, among others.

How to improve: Have reliable linguists and discuss with them. Instead of emojis, consider using your chatbot’s images with different facial expressions or body gestures.

Limit Topics Your Bot Responds To

After a failure with Toy (which caused controversy with its racist and unpleasant comments, leaving the company with no choice but to phase it out), Microsoft released a new bot, Zo, which refuses to converse about politics.

Zo refuses to discuss political topics and responds,

“People can say some awful things when talking politics so I do not discuss.”

Consult your linguists on topics that should be avoided in the countries you’re targeting to avoid the fate of Toy.

Some more suggestions on scripting a conversational polyglot bot:

• Limit jargon, humor, sarcasm, and similar witty language: Those are not translation-friendly and often doesn’t translate well. However, adding humor and wit will help to avoid the robotspeak and make your chatbot sound more human-like. So, if you want to include jokes or puns, make sure they are at least understandable and suitable in the countries you are targeting
• Limit pop culture references, memes: They might not be familiar for users from other countries; or tell linguists to replace those with relevant memes from their cultures. When in Rome, do as the Romans do
• Ban gendered pronouns from bot’s speech: This is unless you know if the person on the other end of the conversation is a man or woman. For example, X.ai’s scheduling bot is programmed not to use any gendered pronouns because its makers “don’t want to burn a bridge with anyone by calling them a he when you’re talking to a woman, or vice versa”
• Remind linguists to be as concise as possible because translated text will expand: No one likes to read walls of text. The bot should speak in single digestible sentences
• Remind linguists to keep the character count low: As a chatbot lives within a chat, every character counts: Try to use short 1–2-syllable words and ask the linguists to do the same where possible
• Avoid complicated grammar structures, better simple than failing.
• Further reading by John Saito of Dropbox on localizable UX writing and designing words.

# LOCALIZING PERSONALITY

More chat-y, less bot-y. 

— XandraLabs slogan

Your bot might be working and able to carry on quite extended conversations, yet users might not want to reengage with it.

Giving it a personality will up your game. Of course, not all bots will need personalities, but having it should help improve retention (something that bots are struggling with) and virality.

It’s important to give your bot a proper, foreign-friendly name; that is a name which is not associated with anything uncool or downright offensive in the countries you localize for. 

You know this localization tale about Chevy Nova, perpetuated by many translation vendors and marketers? As claimed, Nova translates into Spanish as “no go” or “doesn’t go” which spoiled sales for GM in Spanish countries. This “tale” has been debunked, but still serves as a good reminder.

Consider creating different bot personas for different countries but remember about cultural differences.

For example, when attributing some animal image to your bot (metaphorizing), take into account that in different countries there are different connotations associated with different animals.

Or when targeting South Asian audiences, you might want to soften the bot’s image and adjust emojis and visuals (there are studies on this difference in localizing for the East and West).

Share with localizers all the details about your bot personas and their personalities and remind localizers to retain them.

Tone and Empathy Matter

The Guardian found people can be turned off if the experience is too robot-like or casual, so they opted for a polite tone. X.ai found empathy and compassion help their AI assistant, Amy, achieve her goal (getting the meeting scheduled).

Besides sounding less “bot-y,” your bot’s tone should match your company’s voice or simply be suitable for your bot persona. This shouldn’t be lost or altered during the localization — remind your localizers to keep the right tone.

A bot for a legal company shouldn’t speak like a bro. And a bot giving bar recommendations shouldn’t speak like a corporate consultant.

Similar to the traditional app development practice, you may use different tone depending on the message type (e.g. system messages vs. chatting).

Create a styleguide, or “toneguide,” and give it to your foreign wordsmiths.

Let them know who your ideal audience is: They should already be familiar with how that audience communicates in their countries and should imitate that when crafting the script. If not, you might want to instruct them to study conversations between people that represent your target audience prior to writing or localizing your script.

Localize for Human Emotion

Designing for human emotion is a well-known guideline in the dev and design world. This is something you already are doing as a bot dev, but it is as important that your linguists follow it when writing or localizing the script.

Personality is the mysterious force that attracts us to certain people and repels us from others.

 — Aarron Walter, source

When done right, you bot’s personality will evoke emotions and thus will help keep users engaged. The more natural it feels, the more enjoyable the experience is.

For example, Amazon’s Alexa has very human “hmm’s” and “um’s” in her responses. Apple’s Siri assistant knows how to crack a joke and can beatbox upon request.

The personality your bot exudes is what will be upping it’s rankings and virality.

The job of your linguists is to correctly translate that personality and such little traits like above for users in other countries making sure all is as relevant and engaging.

If your team does a good job, people might be actually saying “thanks” to your bot like to a human.

Some good resources on developing for localization:

• Twelve Commandments of Software Localization by Zack Grossbart, Smashing Magazine
• Build Apps for the World by Apple
• IBM Globalization Design Guidelines
• MDN Localization Content Best Practices

Hope you enjoyed it! In the second part, I share tips on collaboration and choosing the right partners for your multilingual bot project, and why bother localizing your bot at all — benefits of localization.

If you liked this post, please click 💚 below. And if you digged it and would like to be notified about my next one, hit subscribe. Also, join my brand-new Polyglot Bots Facebook group here.