Improve search results on Google

how google search results are displayed and how google optimize search engine results and how to get better search results on google
RichardGibbs Profile Pic
RichardGibbs,United Kingdom,Professional
Published Date:04-08-2017
Your Website URL(Optional)
Comment
Search Engine Basics In this chapter, we will begin to explore how search engines work. Building a strong foundation on this topic is essential to understanding the SEO practitioner’s craft. As we discussed in Chapter 1, people have become accustomed to receiving nearly instantaneous answers from search engines after they have submitted a search query. In Chapter 1 we also discussed the volume of queries (more than 7,500 per second). 1 As early as 2008, Google knew about 1 trillion pages on the Web. At SMX Advanced in Seattle in 2014, Google’s Gary Illyes stated that Google now knows about 30,000 trillion pages on the Web. The scale of the Internet/Web (sometimes called the Inter- webs) is growing fast Underlying the enormous problem of processing all these pages is the complex nature of the Web itself. Web pages include text, video, images, and more. It’s easy for humans to understand these and to transition seamlessly between them, but software lacks the intelligence we take for granted. This limitation and others affect how search engines understand the web pages they come across. We’ll discuss some of these limi- tations in this chapter. Of course, this is an ever-changing landscape. The search engines continuously invest in improving their ability to process the content of web pages. For example, advances in image and video search have enabled search engines to inch closer to human-like understanding, a topic that will be explored more in the section “Vertical Search Engines” on page 122. 67Understanding Search Engine Results In the search marketing field, the pages the engines return to fulfill a query are referred to as search engine results pages (SERPs). Each engine returns results in a slightly different format, and these may include vertical results—results that can be derived from different data sources or presented on the results page in a different format, which we’ll illustrate shortly. Understanding the Layout of Search Results Pages Figure 2-1 shows the SERPs in Google for the query stuffed animals. Figure 2-1. Layout of Google search results The various sections outlined in the Google search results are as follows: • Search query box (1) • Vertical navigation (2) • Results information (3) • PPC advertising (4) • Google product search results (5) CHAPTER TWO: SEARCH ENGINE BASICS 68 www.itbooksh ub.com www.cncmanu al.com• Natural/organic/algorithmic results (6) Even though Yahoo no longer does its own crawl of the Web or provides its own search results information (it sources them from Bing), it does format the output dif- ferently. Figure 2-2 shows Yahoo’s results for the same query. Figure 2-2. Layout of Yahoo search results The sections in the Yahoo results are as follows: • Vertical navigation (1) • Search query box (2) • Horizontal navigation (3) • PPC advertising (4) • Natural/organic/algorithmic results (5) • Navigation to more advertising (6) Figure 2-3 shows the layout of the results from Microsoft’s Bing for stuffed animals. UNDERSTANDING SEARCH ENGINE RESULTS 69Figure 2-3. Layout of Bing search results The sections in Bing’s search results are as follows: • Vertical navigation (1) • Search query box (2) • Results information (3) • Time-based refinement options (4) • PPC advertising (5) • Natural/organic/algorithmic results (6) • Query refinement options (7) Each unique section represents a snippet of information provided by the engines. Here are the definitions of what each piece is meant to provide: Vertical navigation Each engine offers the option to search different verticals, such as images, news, video, or maps. Following these links will result in a query with a more limited CHAPTER TWO: SEARCH ENGINE BASICS 70index. In Figure 2-3, for example, you might be able to see news items about stuf- fed animals or videos featuring stuffed animals. Horizontal navigation All three engines used to have some form of horizontal navigation, but as of June 2015 only Yahoo continues to include it. Search query box All of the engines show the query you’ve performed and allow you to edit or reenter a new query from the search results page. If you begin typing, you may notice that Google gives you a list of suggested searches below. This is the Google autocomplete suggestions feature, and it can be incredibly useful for targeting key- words. Next to the search query box, the engines also offer links to the advanced search page, the features of which we’ll discuss later in the book. In addition, you will also see a microphone icon in the right of the search box that allows you to speak your query. In Google image search, this shows up as a camera icon that allows you to upload an image or get similar images back. Results information This section provides a small amount of meta-information about the results that you’re viewing, including an estimate of the number of pages relevant to that par- ticular query (these numbers can be, and frequently are, wildly inaccurate and should be used only as a rough comparative measure). PPC (a.k.a. paid search) advertising The text ads are purchased by companies that use either Google AdWords or Bing. The results are ordered by a variety of factors, including relevance (for which click-through rate, use of searched keywords in the ad, and relevance of the land- ing page are factors in Google) and bid amount (the ads require a maximum bid, which is then compared against other advertisers’ bids). Natural/organic/algorithmic results These results are pulled from the search engines’ primary indices of the Web and ranked in order of relevance and importance according to their complex algo- rithms. This area of the results is the primary focus of this section of the book. Query refinement suggestions Query refinements are offered by Google, Bing, and Yahoo. The goal of these links is to let users search with a more specific and possibly more relevant query that will satisfy their intent. In March 2009, Google enhanced the refinements by implementing Orion Tech- nology, based on technology Google acquired in 2006. The goal of this enhance- ment is to provide a wider array of refinement choices. For example, a search on UNDERSTANDING SEARCH ENGINE RESULTS 71principles of physics may display refinements for the Big Bang, angular momentum, quantum physics, and special relativity. Navigation to more advertising Only Yahoo shows this in the search results. Clicking on these links will bring you to additional paid search results related to the original query. Be aware that the SERPs are always changing as the engines test new formats and lay- outs. Thus, the images in Figure 2-1 through Figure 2-3 may be accurate for only a few weeks or months until Google, Yahoo, and Bing shift to new formats. Understanding How Vertical Results Fit into the SERPs These “standard” results, however, are certainly not all that the engines have to offer. For many types of queries, search engines show vertical results, or instant answers, and include more than just links to other sites to help answer a user’s questions. These types of results present many additional challenges and opportunities for the SEO practitioner. Figure 2-4 shows an example of these types of results. The query in Figure 2-4 brings back a business listing showing an address and the option to get directions. This result attempts to provide the user with the answer he is seeking directly in the search results. Figure 2-4. Local search result for a business CHAPTER TWO: SEARCH ENGINE BASICS 72Figure 2-5 shows another example. The Google search in Figure 2-5 for weather plus a city name returns a direct answer. Once again, the user may not even need to click on a website if all she wanted to know was the temperature. Figure 2-5. Weather search on Google Figure 2-6 is an example of a search for a well-known painter. A Google search for the famous painter Edward Hopper returns image results of some of his most memorable works (shown in the lower-right of the screenshot). This example is a little different from the “instant answers” type of result shown in Figure 2-4 and Figure 2-5. If the user is interested in the first painting shown, he may well click on it to see the paint- ing in a larger size or to get more information about it. For the SEO practitioner, get- ting placed in this vertical result could be a significant win. UNDERSTANDING SEARCH ENGINE RESULTS 73Figure 2-6. Google search on an artist’s name Figure 2-7 shows an example from Yahoo. A query on Yahoo for chicago restaurants brings back a list of popular dining establishments from Yahoo’s local portal. High placement in these results has likely been a good thing for Lou Malnati’s Pizzeria. Figure 2-8 is an example of a celebrity search on Bing. The results in Figure 2-8 include a series of images of the famous actor Charlie Chap- lin. As a last example, Figure 2-9 is a look at the Bing search results for videos with Megan Fox. CHAPTER TWO: SEARCH ENGINE BASICS 74Figure 2-7. Yahoo search for Chicago restaurants At the top of the search results in Figure 2-9, you’re provided with a series of popular videos. Click on a video in the results, and it begins playing right there in the search results. As you can see, the vast variety of vertical integration into search results means that for many popular queries you can expect to receive significant amounts of information in the SERPs themselves. Engines are competing by providing more relevant results and more targeted responses to queries that they feel are best answered by vertical results, rather than web results. UNDERSTANDING SEARCH ENGINE RESULTS 75Figure 2-8. Bing result for Charlie Chaplin As a direct consequence, site owners and web marketers must take into account how this incorporation of vertical search results may impact their rankings and traffic. For many of the searches shown in the previous figures, a high ranking—even in position 1 or 2 in the algorithmic/organic results—may not produce much traffic because of the presentation of the vertical results above them. CHAPTER TWO: SEARCH ENGINE BASICS 76Figure 2-9. Bing result for Megan Fox videos The vertical results also signify an opportunity, as listings are available in services from images to local search to news and products. We will cover how to get included in these results in Chapter 11. Google’s Knowledge Graph The search engines are actively building structured databases of information that allow them to show answers to questions that are not simply links to web pages. In Figure 2-6, the information on the upper right is an example of this. Google provides direct answers in the result, including Edward Hopper’s birth date, place of birth, the date and place of his death, his spouse, and more. In Figure 2-9, Bing provides similar information for Megan Fox. Not only is additional information shown, but it is not just a data dump: it shows that the search engines are working to develop their own knowledge of the relationships between people and things. In the case of Figure 2-6, we can see that Google under- stands that: UNDERSTANDING SEARCH ENGINE RESULTS 77• Edward Hopper is the name of a person. • People have dates and places of birth. • People have dates and places of death. • People might have spouses. The search engines are actively mapping these types of relationships as part of their effort to offer more complete information directly in the search results themselves. Algorithm-Based Ranking Systems: Crawling, Indexing, and Ranking Understanding how crawling, indexing, and ranking works is useful to SEO practition- ers, as it helps them determine what actions to take to meet their goals. This section primarily covers the way Google and Bing operate, and does not necessarily apply to other search engines that are popular in other countries, such as Yandex (Russia), Baidu (China), Seznam (Czech Republic), and Naver (Korea). The search engines must execute many tasks very well to provide relevant search results. Put simplistically, you can think of these as: • Crawling and indexing trillions of documents (pages and files) on the Web (note that they ignore pages that they consider to be “insignificant,” perhaps because the pages are perceived as adding no new value or are not referenced at all on the Web). • Responding to user queries by providing lists of relevant pages. In this section, we’ll walk through the basics of these functions from a nontechnical perspective. This section will start by discussing how search engines find and discover content. Crawling and Indexing To offer the best possible results, search engines must attempt to discover all the public pages on the World Wide Web and then present the ones that best match up with the user’s search query. The first step in this process is crawling the Web. The search engines start with a seed set of sites that are known to be very high quality, and then visit the links on each page of those sites to discover other web pages. The link structure of the Web serves to bind together all of the pages that were made public as a result of someone linking to them. Through links, search engines’ automa- ted robots, called crawlers or spiders, can reach the many trillions of interconnected documents. CHAPTER TWO: SEARCH ENGINE BASICS 78In Figure 2-10, you can see the home page of USA.gov, the official U.S. government website. The links on the page are outlined in red. Crawling this page would start with loading the page, analyzing the content, and then seeing what other pages USA.gov links to. Figure 2-10. Crawling the U.S. government website The search engine would then load those other pages and analyze that content as well. This process repeats over and over again until the crawling process is complete. This process is enormously complex, as the Web is a large and complex place. NOTE Search engines do not attempt to crawl the entire Web every day. In fact, they may become aware of pages that they choose not to crawl because those pages are not likely to be important enough to return in a search result. We will discuss the role of importance in “Retrieval and Ranking” on page 80. The first step in this process is to build an index of terms. This is a massive database that catalogs all the significant terms on each page crawled by the search engine. ALGORITHM-BASED RANKING SYSTEMS: CRAWLING, INDEXING, AND RANKING 79A lot of other data is also recorded, such as a map of all the pages that each page links to, the clickable text of those links (known as the anchor text), whether or not those links are considered ads, and more. To accomplish the monumental task of holding data on hundreds of trillions of pages that can be accessed in a fraction of a second, the search engines have constructed massive data centers to deal with all this data. One key concept in building a search engine is deciding where to begin a crawl of the Web. Although you could theoretically start from many different places on the Web, you would ideally begin your crawl with a trusted seed set of websites. Starting with a known, trusted set of websites enables search engines to measure how much they trust the other websites that they find through the crawling process. We will discuss the role of trust in search algorithms in more detail in “How Links Histori- cally Influenced Search Engine Rankings” on page 421. Retrieval and Ranking For most searchers, the quest for an answer begins as shown in Figure 2-11. Figure 2-11. Start of a user’s search quest The next step in this quest occurs when the search engine returns a list of relevant pages on the Web in the order it believes is most likely to satisfy the user. This process requires the search engines to scour their corpus of hundreds of billions of documents and do two things: first, return only the results that are related to the searcher’s query; and second, rank the results in order of perceived importance (taking into account the trust and authority associated with the site). It is both relevance and importance that the process of SEO is meant to influence. Relevance is the degree to which the content of the documents returned in a search matches the user’s query intention and terms. The relevance of a document increases if the page contains terms relevant to the phrase queried by the user, or if links to the page come from relevant pages and use relevant anchor text. You can think of relevance as the first step to being “in the game.” If you are not rele- vant to a query, the search engine does not consider you for inclusion in the search results for that query. We will discuss how relevance is determined in more detail in “Determining Searcher Intent and Delivering Relevant, Fresh Content” on page 92. Importance refers to the relative importance, measured via citation (the act of one work referencing another, as often occurs in academic and business documents), of a given document that matches the user’s query. The importance of a given document increa- CHAPTER TWO: SEARCH ENGINE BASICS 80ses with every other document that references it. In today’s online environment, cita- tions can come in the form of links to the document or references to it on social media sites. Determining how to weight these signals is known as citation analysis. You can think of importance as a way to determine which page, from a group of equally relevant pages, shows up first in the search results, which is second, and so forth. The relative authority of the site, and the trust the search engine has in it, are significant parts of this determination. Of course, the equation is a bit more complex than this, and not all pages are equally relevant. Ultimately, it is the combination of relevance and importance that determines the ranking order. So, when you see a search results page such as the one shown in Figure 2-12, you can surmise that the search engine (in this case, Bing) believes the Superhero Stamps page on eBay has the highest combined score for relevance and importance for the query marvel superhero stamps. Figure 2-12. Sample search result for “marvel superhero stamps” Importance and relevance aren’t determined manually (those trillions of man-hours would require Earth’s entire population as a workforce). Instead, the engines craft careful, mathematical equations—algorithms—to sort the wheat from the chaff and then rank the wheat in order of quality. These algorithms often comprise hundreds of components. In the search marketing field, they are often referred to as ranking factors or algorithmic ranking criteria. We discuss ranking factors or signals (signals is the term Google prefers) in more detail in “Analyzing Ranking Factors” on page 108. ALGORITHM-BASED RANKING SYSTEMS: CRAWLING, INDEXING, AND RANKING 81Evaluating Content on a Web Page Search engines place a lot of weight on the content of each web page. After all, it is this content that defines what a page is about, and the search engines do a detailed analysis of each web page they find during their crawl to help make that determination. You can think of this as the search engine performing a detailed analysis of all the words and phrases that appear on a web page, and then building a map of that data for it to consider showing your page in the results when a user enters a related search query. This map, often referred to as a semantic map, seeks to define the relationships between those concepts so that the search engine can better understand how to match the right web pages with user search queries. If there is no semantic match of the content of a web page to the query, the page has a much lower possibility of showing up. Therefore, the words you put on the page, and the “theme” of that page, play a huge role in ranking. Figure 2-13 shows how a search engine will break up a page when it looks at it, using a page on the Forbes website. The navigational elements of a web page are likely similar across the many pages of a site. These navigational elements are not ignored, and they do play an important role, but they do not help a search engine determine what the unique content is on a page. To do that, the search engine focuses on the part of Figure 2-13 that is labeled “Unique Page Content.” Determining the unique content on a page is an important part of what the search engine does. The search engine uses its understanding of unique content to determine the types of search queries for which the web page might be relevant. Because site navigation is generally not unique to a single web page, it does not help the search engine with that task. CHAPTER TWO: SEARCH ENGINE BASICS 82Figure 2-13. Breaking up a web page This does not mean navigation links are not important—they most certainly are; how- ever, they simply do not count when a search engine is trying to determine the unique content of a web page, as they are shared among many web pages. One task the search engines face is judging the value of content. Although evaluating how the community responds to a piece of content using link analysis is part of the process, the search engines can also draw some conclusions based on what they see on the page. For example, is the exact same content available on another website? Is the unique content the search engine can see two sentences long or 500 words long? Does the content repeat the same keywords excessively? These are a few examples of factors the search engine can evaluate when trying to determine the value of a piece of content. Understanding What Content Search Engines Can “See” on a Web Page Search engine crawlers and indexing programs are basically software programs. These programs are extraordinarily powerful. They crawl hundreds of trillions of web pages, analyze the content of all these pages, and analyze the way all these pages link to one another. Then they organize this into a series of databases that can respond to a user search query with a highly tuned set of results in a few tenths of a second. This is an amazing accomplishment, but it has its limitations. Software is very mechan- ical, and it can understand only portions of most web pages. The search engine crawler ALGORITHM-BASED RANKING SYSTEMS: CRAWLING, INDEXING, AND RANKING 83analyzes the raw HTML form of a web page. If you want to see what this looks like, you can do so by using your browser to view the source. Figure 2-14 shows how to do that in Chrome, and Figure 2-15 shows how to do that in Firefox. Typically you can access it most easily by right-clicking with your mouse on a web page to access a hidden menu. Figure 2-14. Viewing source in Chrome: right-click on the web page to access the menu CHAPTER TWO: SEARCH ENGINE BASICS 84Figure 2-15. Viewing source in Firefox There are also various in-browser web development tools (add-ons and extensions) that facilitate viewing source code in your browser of choice, as well as detecting web applications and JavaScript libraries. One of the most widely used code analysis tools is Web Developer, by Chris Pederick, available for Chrome, Firefox, and Opera. Once you view the source, you will be presented with the exact code for the web page that the web server sent to your browser. This is most of what the search engine crawler sees (the search engine also sees the HTTP headers for the page, which are status codes it receives from the web server where the page is hosted). In Some cases Google will execute JavaScript on the page as well. For more on how they do this, please refer to Chapter 6. When trying to analyze the user-visible content on a web page, search engines largely ignore code related to the navigation and display of the page, such as that shown in Figure 2-16, as it has nothing to do with the page’s content. ALGORITHM-BASED RANKING SYSTEMS: CRAWLING, INDEXING, AND RANKING 85Figure 2-16. Sample web page source code The search engine crawler is most interested in the HTML text on the page. Figure 2-17 is an example of HTML text for the Moz home page. Figure 2-17. Sample HTML text in the source code showing real content Although Figure 2-17 still shows some HTML encoding, you can see the “regular” text clearly in the code. This is the unique content that the crawler is looking to find. In addition, search engines read a few other elements. One of these is the page title. The page title is one of the most important factors in ranking a given web page. It is the text that shows in the browser’s title bar (above the browser menu and the address bar). Figure 2-18 shows the code that the crawler sees, using Trip Advisor as an example. The first highlighted area in Figure 2-18 is for the title tag. The title tag is also often (but not always) used as the title of your listing in search engine results (see Figure 2-19). CHAPTER TWO: SEARCH ENGINE BASICS 86

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.