How to Index your site pages posts and content

Indexing the pages and posts on your site is the best way to tell Search Engine robots which content to take notice of, and which content should be ignored.

Learn how to index your pages and posts right with this step by step guide.

What is indexing?

When search engine robots scour your site, it’s called ‘indexing’. Essentially, robots act like librarians, sweeping up all of the content on the web, cataloguing and indexing it, so it can be served back in search results.

Everything held within your website is (potentially) visible to search engine bots and able to be crawled: Media (images video, pdfs); pages and blog posts.

This is fine if you’re meticulous and diligent at only uploading the beautiful bare essentials to your site. If it’s not pin perfect and robot ready, make sure you’ve set the content you don’t want to be shown to the world, to ‘no-index’

And now for the embarassing story:

Awkward! Until recently, I had an old page filled with random content in this site. I’d titled the page ‘Care Plans’, but had failed to fill the page with real information. Built on an old theme, the only content this orphan page held was very unattractive short coding.

When a potential client recently googled to see if Love Communications website care plans were the right fit for him, Google served this page. Needless to say, a useless, indicipherable page full of code scratchings did not seal the deal.

I’d failed to set this page to no-index, and lost a client as a consequence.

Custard on my face! Bank not in balance. Expensive and excruciating lesson well learnt.

How to index (make visible) or no-index (hide) the content on your site?

To hide specific pages, posts and media from search bots, you have to set them in to ‘no-index’.

1. Use YOAST SEO plugin
   • after writing your post or page, scroll down to the Yoast meta box.
   • Here, fill in your main keyword/keyword phrase, ‘slug’ (the keyworded url) and keyworded meta description
   • Scroll further until you get to the ‘advanced’ field.
   • Choose your setting via the drop down menu – either index (allow search engines to show this in search results) or, no-index (don’t show in search results)

Protecting PDFs

To make sure any media on your site you’d like to hide from public view stays protected, use robot.txt

Using Yoast SEO plugin:

1. Click on the ‘Tools’ field
2. Choose ‘create robots.txt file’
3. Add a ‘disallow’ rule to set the pdf files as no-index
4. Put this code in the Disallow: line *.pdf
5. This code tells indexing robots that all the pdf files are to be excluded from search results

To disallow a page, add another disallow line, and use this protocol – Disallow: /pageurl (no need to include the root domain)

I use the free version of YOAST SEO to manage these indexing changes. You can use any other SEO plugins that have this ability.