App IndexingGoogleMarketingTechnical

Creating an App Content Sitemap for Google App Indexing

By March 23, 2016 No Comments

Eighty-nine percent of smartphone users start with Google when they want to find something on the Internet. And yet, an average user who spends three and a half hours on their phone each day is inside apps for all but 20 minutes of that time. This is a huge disconnect, and despite the best efforts of Apple and Google, searching app content on mobile devices is often unproductive.

As a mobile app developer, this is a major problem. Fortunately, we at Branch have discovered a simple solution that uses well-established standards already supported by Google. We’re calling it App Content Sitemaps.

What are App Content Sitemaps, and why would I want one?

So why would you want to bring Google into your closed application? The answer is simple: because you want traffic, and Google search results provide that traffic. Your product doesn’t exist if a user doesn’t see your application — or content from your application — when they’re searching for a solution. In most cases, that’s why we leverage Search Engine Optimization (SEO). But SEO in apps is totally different — and increasingly important.

Doesn’t Google already do this?

No, Google does not index your app’s content automatically. Yes, the tech giant would REALLY love to do exactly that, but it’s difficult to search content inside apps. Difficult enough that the world’s leading search engine is still puzzling over this problem. Why? It has to do with how Google discovers content on the web.

Right now, Google leverages two main approaches to how it discovers content on public websites:

  1. Organic crawling — when Googlebot (Google’s web crawling bot) happens to stumble upon a link to your content from somewhere else.
  2. Sitemaps — when you intentionally submit webpages for Googlebot to index.

Both methods have something in common: they rely on Googlebot’s access to the public web. That’s a problem for app content because app content is trapped inside an app; it’s not on the public web.  

Google is working to help Googlebot crawl app content through a collection of initiatives dubbed “App Indexing.” Google isn’t there yet, but it’s on its way. For now, the only real solution is to tie your pieces of content on the web to equivalent pieces of content inside your application. When you do this, you can automatically route users into your app, but only after they have it installed. In other words: Google trusts that your website, and the references leading into your app on your website, are what you promise them to be.

So what does this mean? This means you can adapt the two traditional approaches to content discovery to app content. All you have to do is to define the relationship between website and app. This is easier said than done. Indexing your app for Google is confusing and labor intensive. To top it off, Google gives you little to no feedback on the success of your efforts. Regardless, this is probably the best you can do if you want incoming traffic.

Okay, how do I create an App Content Sitemap?

If you use Branch for your deep linking, skip down about two pages. All of this work has already been done for you. Otherwise, keep reading:

Step 1: Configure your app to support deep link routing

Make sure your deep links work. If a visitor clicks on a link for a particular piece of content, then that piece of content must be delivered. Otherwise, your users will assume your app is broken.

Step 2: Create a web version of your app content

Assume every piece of content in your app needs to have a corresponding web version. Why? So Googlebot can access it. Remember: your website content (not the content inside your app) is what Googlebot indexes, so make sure it’s accurate and reflects best practices for mobile web SEO.

Step 3: Verify your site with Google

No matter what you do your website will need to be verified. This helps Google confirm your ownership of the domain and its legitimacy. The steps are easy:

  1. Use Google Search Console to configure a website property.
  2. Use one of the verification methods on this page. Important: please note the instructions marked “Verify a mobile app” on this page are not necessary for app content sitemaps in most situations. We know, counterintuitive.

Step 4: Enable App Indexing

Here’s where the magic happens: when properly configured, search result links will immediately launch your application (assuming the user has installed your app) instead of pushing them to your website. It’s a wonderful, fantastic, grand and immersive experience when it works.

AllTheCooks App Indexing

You could skip this step by redirecting a user after they arrive on the web version of your content. But there are two problems with this approach: first, it’s a less user-friendly experience; second, Google won’t give you the “App Results” treatment they give to webpages that tie themselves with apps. So let’s take a look at how to do that with two of the biggest ecosystems out there: iOS and Android.

iOS configuration

When it comes to iOS, you need to familiarize yourself with Universal Links (something we recommend for all iOS apps). Universal Links can be complicated. Complicated enough that we can’t explain it here. So, if you’re not already familiar with Universal Links, you can check out our blog post explaining the whole scheme.

You can optionally also add a <link> tag in the header of each page of content on your website to indicate deep link information:

<link rel="alternate" href="ios-app://app_store_app_id/your_app_scheme/blog/post12345">

Replace app_store_app_id with the ID of your app in the Apple App Store (you can find this by going to the App Store web page for your app, and looking for the digits following id at the end of the URL. For example: https://itunes.apple.com/us/app/your_app_name/id000000000).

Android configuration

Android, by comparison, is simple. All you need to do is configure your app to respond to a custom URI scheme. We’ve already done a good deal of the work for you. Simply paste the following intent filter inside the <activity></activity> tags for the Activity you want to launch when the search result is opened:

<intent-filter>
    <data android:scheme="your_app_scheme"/>
    <action android:name="android.intent.action.VIEW" />
    <category android:name="android.intent.category.DEFAULT" />
    <category android:name="android.intent.category.BROWSABLE" />
</intent-filter>

Replace your_app_scheme with something of your own selection (it should be unique to your app).

Next, use a <link> tag in the header of each page of content on your website to indicate deep link information:

<link rel="alternate" href="android-app://your.package.name/your_app_scheme/blog/post12345">

In this example, we assume the following:

  • Package name: your.package.name
  • URI scheme: your_app_scheme
  • Deep link path: blog/post12345

Note: you may also use HTTP links instead of custom URI scheme links, as described in these steps from Google. This approach requires the extra instructions marked “Verify a mobile app” mentioned above, and may result in your users being asked to select between using your app and any installed web browsers when opening your links.

Once your website is up and App Indexing is configured, Google will (in theory) begin associating your web content with your app content in search results. However, this only works if Google can find your content in the first place. Unlike traditional websites, which have always lived in the land of easily-sharable URLs, apps are a closed system. Googlebot doesn’t easily index closed systems. But that’s fixable with App Content Sitemaps.

An App Content Sitemap allows you to manually submit an index of your pages to Google. Please note: when you do this Google will know that your app content exists, but it may not actually index your content. It’s still a valuable thing to do.

Step 5: Generate a sitemap of your site

So how do you generate a sitemap? For our purposes, it’s actually pretty difficult to do. Not only do you need to generate a large XML document, but you also need to keep it up-to-date and in sync with your app’s deep links. How you do this will depend on your system configuration (unless Branch is doing this for you), but the end result needs to look something like this: 

<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://your-website.com/path/to/awesome/content</loc>
   </url>
   <url>
      <loc>https://your-website.com/path/to/awesome/content-2</loc>
   </url>
   <url>
      <loc>https://your-website.com/path/to/awesome/content-3</loc>
   </url>
</urlset>

We have built a handly App Indexing Validator based on what we have seen working in the wild. You can use this tool to help verify whether your configuration is working correctly.

Step 6: Upload this sitemap to Google Search Console

Now all you have to do is tell Google that you’ve a sitemap for them to use. You can upload the file directly to the Search Console, or simply add these lines to your robots.txt:

User-agent: *
Sitemap: https://your-website.com/sitemap.xml

Step 7: Wait

Last, but not least, you’ll need to wait. How Google’s search algorithm implements app content is nothing short of opaque, even at the best of times. We know that app content does appear in search results when submitted using the above process, but many of the variables at work are still unclear.

Mobile App Indexing Example

What is Branch doing to make this easier?

For Branch users, we already know all about your content and where it lives inside your app. We automatically create web placeholders for each piece of in-app content with every link your users generate. We handle the standards, the sitemaps, and the content needing to sync with Google. We even know when different links point to the same piece of content, so you don’t have to worry about multiple copies. All you need to do is check a box and Branch handles the rest.

Branch App Indexing Setup

We can help you see what’s working

Most importantly, we give you quantifiable insights into what’s happening behind the scenes. We measure sitemap generation for application content, how much unique content was included, and how many pieces of content Google actually crawled.

Branch App Indexing Status

Our links analytics tool can help you track traffic and installs through your sitemap-submitted links with just one filter. These results are already showing up in search results and confirm that install and open analytics work as expected:

App Indexing Performance and Stats

How does this work if I already have a website of my own?

Highly-ranked websites may not want bnc.it to appear in links next to their content. We’re working on it. Currently, we’re moving to integrate with custom domains and look forward to sharing an update with you soon.

The search field is changing

We mentioned eighty-nine percent of smartphone users start with Google when they want to find something on the Internet. What you probably don’t realize is that mobile device searches greatly outnumber searches from computers. That means content discovery will need to adjust, regardless of the ecosystem. We feel in-app content in search results is the next big issue Google needs to solve. While we know submitting app content via sitemaps works, something needs to change before it becomes equal to content from traditional websites with high domain scores. No matter what happens next, you can count on Branch to help.


 

Additional Resources

https://developers.google.com/app-indexing/?hl=en

https://support.google.com/webmasters/answer/183668

https://support.google.com/webmasters/answer/156184?hl=en&ref_topic=4581190

https://support.google.com/googleplay/android-developer/answer/6041489

https://support.google.com/webmasters/answer/139066

http://developer.android.com/intl/in/training/app-indexing/enabling-app-indexing.html