Microbrowsers Are Everywhere

Colin Bendell gets into the minutia of microbrowsers - the small previews of your site that are pervasive all around the web and through social media apps and search engines whenever an item of content on your site is referenced.

Youve seen it everywhere - that little thumbnail preview of a website mentioned in a tweet, the expanded description in a Slack channel, or in WhatsApp group chat.

These link previews are so commonplace that we hardly pay any attention to how our site design might be impacting the generated preview. Yet, these previews can be the most influential part for attracting new audiences and increasing engagement - possibly more than SEO. Even more alarming is that most web analytics are blind to this traffic and cant show you how these Microbrowsers are interacting with your site.

As we close out the year, here are five essential questions and ideas that every web dev should know about Microbrowsers.

1. What are Microbrowsers? How are they different from normal browser?

We are all very familiar with the main browsers like Firefox, Safari, Chrome, Edge and Internet Explorer. Not to mention the many new browsers that use Chromium as the rendering engine but offer unique user experiences like Samsung Internet or Brave.

In contrast, Microbrowsers are a class of User-Agents that also visit website links, parse HTML and generate a user experience. But unlike those traditional browsers, the HTML parsing is limited and the rendering engine is singularly focused. The experience is not intended to be interactive. Rather the experience is intended to be representational - to give the user a hint of what exists on the other side of the URL.

Creating link previews is not new. Facebook and Twitter have been adding these link previews in posts for nearly a decade. That used to be the primary use case. Marketing teams created backlog items to adopt different micro data - from Twitter Cards and Open Graph annotations for Facebook. LinkedIn likewise embraced both Open Graph and OEmbed tags to help generate the previews

<meta name="description" content="seo description long"><meta name="keywords" content="seo keyword list"><link rel="shortcut icon" href="favicon.ico"                           type="image/x-icon"><link rel="icon" href="favicon_32.png" sizes="32x32"><link rel="icon" href="favicon_48.png" sizes="48x48"><link rel="icon" href="favicon_96.png" sizes="96x96"><link rel="icon" href="favicon_144.png" sizes="144x144"><meta property="og:title" content="Short title here" /><meta property="og:description" content="shortish description" /><meta name="twitter:title" content="Short title here"><meta name="twitter:description" content="shortish description"><meta property="og:image"      content="https://res.cloudinary.com/.../hero-img.png" /><meta name="twitter:image:src"      content="https://res.cloudinary.com/.../hero-img.png">

As group chats and other collaboration tools have become more prevalent, we have seen many features from the big social media platforms emerge. Particularly in recent years weve seen the adoption of the link unfurling behaviour in these chat platforms. Rather than reinventing the wheel, each platform looks for pre-existing micro data to generate the preview.

The challenge now is that each communication platform parses and interprets this micro-data in different ways, and presents the information in differing ways.

2. If Microbrowsers are everywhere, why dont I see them in my analytics reports?

Its easy to miss the traffic from Microbrowsers. This is for a number of reasons:

First, page requests from Microbrowsers dont run JavaScript and they dont accept cookies. The Google Analytics <script> block wont be run or executed. And all cookie will be ignored by the rendering agent.

Second, if you were to do a log analysis based on HTTP logs from your CDN or web stack, you would see a relatively small volume of traffic. That is assuming you can identify the User-Agent strings. Some of these Microbrowsers impersonate real browsers and others impersonate Facebook or twitter. For example, iMessage uses the same User-Agent string for all these requests and it hasnt changed since iOS 9.

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)              AppleWebKit/601.2.4 (KHTML, like Gecko)              Version/9.0.1 Safari/601.2.4              facebookexternalhit/1.1               Facebot Twitterbot/1.0

Finally, many platforms - particularly Facebook Messenger and Hangouts use centralized services to request the preview layout. This is in contrast to WhatsApp* and iMessage where you will see one request per user. In the centralized consumer approach your web servers will only see one request, but this one request might represent thousands of eyeballs.

3. Microbrowser are probably more important than google bot

We all know the importance of having our web sites crawled by search engines like googlebot. These bots are the lifeblood for lead generation and for discovering new users.

However, the real gold for marketers is from word-of-mouth discussions. Those conversations with your friends when you recommend a TV show, a brand of clothing, or share a news report. This is the most valuable kind of marketing.

Last year when assembling the data for Cloudinarys State of the Visual Media report, I discovered that there was a very prominent usage pattern over the USA holiday season. During thanksgiving, all the way to Black Friday, the rate of link sharing sky rocketed as group chats shared deals and insights.

Zooming out (and normalizing for time-of-day), we can see that there is a daily cadence of link sharing and word of mouth referrals. It probably isnt a shock to see that we predominantly share links in Slack between Monday and Friday, while WhatsApp is used all week long. Likewise, WhatsApp is most often used during our break times like lunch or in the evening after we put the kids to bed.

While the link preview is increasingly common, there are two user behaviours to balance:

• Users can be skeptical of links sent via SMS and other chats. We dont want to be fooled into clicking a fishing links and so we look for other queues to offer validation. This is why most platforms use the preview while also emphasize the website url host name.

• Skimming. Im sure youve had the experience coming out of a meeting or grocery store to find a group chat with 100 messages. As you scroll to catch up on the conversation, links can easily be skipped. In this way, users expect the preview to act as a summary to tell them how important it is to visit the link.

4. Microbrowsers are not real browsers (they just play one on TV)

As I previously mentioned, Microbrowsers pretend to be a browser in that they send the right HTTP headers and often send impersonating User-Agent strings. Yet, there are several characteristics that a web dev should be aware of.

First, Microbrowsers try to protect the Users privacy. The user hasnt decided to visit your site yet, and more importantly, the user is having a private conversation. The fact that your brand or website is mentioned should just make your ears burn, but you shouldnt be able to listen in to the conversation.

For this reason, all Microbrowsers:

• dont execute JavaScript - so your react application wont work
• ignore all cookies - so your A/B or red/green cookies will be ignored
• some will follow redirects, but will quickly time out after a few seconds and give up trying to expand the link.
• there wont be a referer: HTTP header when the user clicks the link for the full browser. In fact, a new user will appear as direct traffic - as though they typed in the url.

Second, Microbrowsers have a very small brain and very likely dont use an advanced network algorithm. Most browsers will use a tokenizer to parse the HTML markup and send requests to the network stack asynchronously. Better yet, browsers will do some analysis of the resources needed before sending the async request to the network.

Based on observational experimentation, most platforms simply use a glorified for loop when parsing the HTML and often request the resources synchronously. This might be ok for fast wifi experiences, but it can cause inconsistent experiences on flaky wifi.

For example, iMessage will discover and load all <link rel="icon" > favicon, all <meta property="og:image" images, and all <meta name="twitter:image:src" before deciding what to render. Many sites still advertise 5 or more favicon sizes. This means that iMessage will download all favicons regardless of size and then not use them if it decides to instead render the image.

For this reason the meta markup that is included is important. The lighter the content, the more likely it will be to be rendered.

5. Markup Matters

Since Microbrowsers are simple-brained browsers, it is all the more important to produce good markup. Here are a few good strategies:

• Its almost 2020, advertise one favicon size. Remove all the other <link rel="shortcut icon" and <link rel="icon" references.

• Based on observational experimentation, the most commonly recognized micro data tags for preview are the Open-Graph tags. When the OG and twitter card tags are missing, the default SEO <meta name="description" is used. However, since the description is often nonsensical SEO optimized phrases, users eyes will likely glaze over.

• On that note, use good descriptive text
• Provide up to three <meta property="og:image" images. Most platforms will only load the first one, while others (notably iMessage) attempts to create a collage.

• Use <meta property="og:video* with progressive (not streaming) video experiences.

<meta property="og:type" content="video.other"><meta property="og:video:url"                content="https://shoesbycolin.com/blue.mp4"><meta property="og:video:secure_url"                content="https://shoesbycolin.com/blue.mp4"><meta property="og:video:type" content="video/mp4"><meta property="og:video:width" content="1280"><meta property="og:video:height" content="720">

• Dont use UA sniffing to hide the <meta> tags. Sites like Amazon do this to try and show only Facebook/Twitter the micro data annotated website. But this can cause problems for some Microbrowsers that dont use the same impersonation convention. The result is a simple link without a preview.
• Use the opportunity to tell your product story or summarize your ideas.

Summary

As more of our conversations happen in group chats and slack channels, link previews are an important way for you to engage users before they start the journey on your site. Unfortunately, not all websites present good or compelling previews. (And now that you know what to look for, you wont be able to unsee bad examples - Im sorry). To help users take the leap and visit your site, we need to make sure that all our pages are annotated with micro data. Better yet, we can use these previews to create compelling visual summaries.

About the author

Colin is part of the CTO Office at Cloudinary and co-author of the OReilly book High Performance Images. He spends much of his time at the intersection of high volume data, media, browsers and standards. He recently helped the community effort writing chapters in the Web Almanac on Media and CDNs. You can find him on tweeting @colinbendell and at blogging at https://bendell.ca

More articles by Colin