Posted by rjonesx.

Here at Moz we have committed to making Link Explorer as similar to Google as possible, specifically in the way we crawl the web. I have discussed in previous articles some metrics we use to ascertain that performance, but today I wanted to spend a little bit of time talking about the impact of robots.txt and crawling the web.

Most of you are familiar with robots.txt as the method by which webmasters can direct Google and other bots to visit only certain pages on the site. Webmasters can be selective, allowing certain bots to visit some pages while denying other bots access to the same. This presents a problem for companies like Moz, Majestic, and Ahrefs: we try to crawl the web like Google, but certain websites deny access to our bots while allowing that access to Googlebot. So, why exactly does this matter?

Why does it matter?

As we crawl the web, if a bot encounters a robots.txt file, they’re blocked from crawling specific content. We can see the links that point to the site, but we’re blind regarding the content of the site itself. We can’t see the outbound links from that site. This leads to an immediate deficiency in the link graph, at least in terms of being similar to Google (if Googlebot is not similarly blocked).

But that isn’t the only issue. There is a cascading failure caused by bots being blocked by robots.txt in the form of crawl prioritization. As a bot crawls the web, it discovers links and has to prioritize which links to crawl next. Let’s say Google finds 100 links and prioritizes the top 50 to crawl. However, a different bot finds those same 100 links, but is blocked by robots.txt from crawling 10 of the top 50 pages. Instead, they’re forced to crawl around those, making them choose a different 50 pages to crawl. This different set of crawled pages will return, of course, a different set of links. In this next round of crawling, Google will not only have a different set they’re allowed to crawl, the set itself will differ because they crawled different pages in the first place.

Long story short, much like the proverbial butterfly that flaps its wings eventually leading to a hurricane, small changes in robots.txt which prevent some bots and allow others ultimately leads to very different results compared to what Google actually sees.

So, how are we doing?

You know I wasn’t going to leave you hanging. Let’s do some research. Let’s analyze the top 1,000,000 websites on the Internet according to Quantcast and determine which bots are blocked, how frequently, and what impact that might have.

Methodology

The methodology is fairly straightforward.

1. Download the Quantcast Top Million
2. Download the robots.txt if available from all top million sites
3. Parse the robots.txt to determine whether the home page and other pages are available
4. Collect link data related to blocked sites
5. Collect total pages on-site related to blocked sites.
6. Report the differences among crawlers.

Total sites blocked

The first and easiest metric to report is the number of sites which block individual crawlers (Moz, Majestic, Ahrefs) while allowing Google. Most site that block one of the major SEO crawlers block them all. They simply formulate robots.txt to allow major search engines while blocking other bot traffic. Lower is better.

Of the sites analyzed, 27,123 blocked MJ12Bot (Majestic), 32,982 blocked Ahrefs, and 25,427 blocked Moz. This means that among the major industry crawlers, Moz is the least likely to be turned away from a site that allows Googlebot. But what does this really mean?

Total RLDs blocked

As discussed previously, one big issue with disparate robots.txt entries is that it stops the flow of PageRank. If Google can see a site, they can pass link equity from referring domains through the site’s outbound domains on to other sites. If a site is blocked by robots.txt, it’s as though the outbound lanes of traffic on all the roads going into the site are blocked. By counting all the inbound lanes of traffic, we can get an idea of the total impact on the link graph. Lower is better.

According to our research, Majestic ran into dead ends on 17,787,118 referring domains, Ahrefs on 20,072,690 and Moz on 16,598,365. Once again, Moz’s robots.txt profile was most similar to that of Google’s. But referring domains isn’t the only issue with which we should be concerned.

Total pages blocked

Most pages on the web only have internal links. Google isn’t interested in creating a link graph — they’re interested in creating a search engine. Thus, a bot designed to act like Google needs to be just as concerned about pages that only receive internal links as they are those that receive external links. Another metric we can measure is the total number of pages that are blocked by using Google’s site: query to estimate the number of pages Google has access to that a different crawler does not. So, how do the competing industry crawlers perform? Lower is better.

Once again, Moz shines on this metric. It’s not just that Moz is blocked by fewer sites— Moz is blocked by less important and smaller sites. Majestic misses the opportunity to crawl 675,381,982 pages, Ahrefs misses 732,871,714 and Moz misses 658,015,885. There’s almost an 80 million-page difference between Ahrefs and Moz just in the top million sites on the web.

Unique sites blocked

Most of the robots.txt disallows facing Moz, Majestic, and Ahrefs are simply blanket blocks of all bots that don’t represent major search engines. However, we can isolate the times when specific bots are named deliberately for exclusion while competitors remain. For example, how many times is Moz blocked while Ahrefs and Majestic are allowed? Which bot is singled out the most? Lower is better.

Ahrefs is singled out by 1201 sites, Majestic by 7152 and Moz by 904. It is understandable that Majestic has been singled out, given that they have been operating a very large link index for many years, a decade or more. It took Moz 10 years to accumulate 904 individual robots.txt blocks, and took Ahrefs 7 years to accumulate 1204. But let me give some examples of why this is important.

If you care about links from name.com, hypermart.net, or eclipse.org, you can’t rely solely on Majestic.

If you care about links from popsugar.com, dict.cc, or bookcrossing.com, you can’t rely solely on Moz.

If you care about links from dailymail.co.uk, patch.com, or getty.edu, you can’t rely solely on Ahrefs.

And regardless of what you do or which provider you use, you can’t links from yelp.com, who.int, or findarticles.com.

Conclusions

While Moz’s crawler DotBot clearly enjoys the closest robots.txt profile to Google among the three major link indexes, there’s still a lot of work to be done. We work very hard on crawler politeness to ensure that we’re not a burden to webmasters, which allows us to crawl the web in a manner more like Google. We will continue to work more to improve our performance across the web and bring to you the best backlink index possible.

Thanks to Dejan SEO for the beautiful link graph used in the header image and Mapt for the initial image used in the diagrams.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by willcritchlow

It should be quite obvious for anyone that knows me that I’m not a lawyer, and therefore that what follows is not legal advice. For anyone who doesn’t know me: I’m not a lawyer, I’m certainly not your lawyer, and what follows is definitely not legal advice.

With that out of the way, I wanted to give you some bits of information that might feed into your GDPR planning, as they come up more from the marketing side than the pure legal interpretation of your obligations and responsibilities under this new legislation. While most legal departments will be considering the direct impacts of the GDPR on their own operations, many might miss the impacts that other companies’ (namely, in this case, Google’s) compliance actions have on your data.

But I might be getting a bit ahead of myself: it’s quite possible that not all of you know what the GDPR is, and why or whether you should care. If you do know what it is, and you just want to get to my opinions, go ahead and skip down the page.

What is the GDPR?

The tweet-length version is that the GDPR (General Data Protection Regulation) is new EU legislation covering data protection and privacy for EU citizens, and it applies to all companies offering goods or services to people in the EU.

Even if you aren’t based in the EU, it applies to your company if you have customers who are, and it has teeth (fines of up to the greater of 4% of global revenue or EUR20m). It comes into force on May 25. You have probably heard about it through the myriad organizations who put you on their email list without asking and are now emailing you to “opt back in.”

In most companies, it will not fall to the marketing team to research everything that has to change and achieve compliance, though it is worth getting up to speed with at least the high-level outline and in particular its requirements around informed consent, which is:

“…any freely given, specific, informed, and unambiguous indication of the data subject’s wishes by which he or she, by a statement or by a clear affirmative action, signifies agreement to the processing of personal data relating to him or her.”

As always, when laws are made about new technology, there are many questions to be resolved, and indeed, jokes to be made:

Can you recommend a GDPR expert?
-yes
Can I have their email address?
-no
— Adam Cleevely (@ACleevely) May 2, 2018

But my post today isn’t about what you should do to get compliant — that’s specific to your circumstances — and a ton has been written about this already:

• I’ve found value in the content Ometria has produced on this front, e.g. 6 things e-commerce marketers should know about GDPR and their deeper GDPR guide (registration required)
• If you work in the area, this GDPR impact on social post from Buffer will get you up to speed there
• And for the paid marketers among us, this GDPR impact on AdWords article from WordStream is what you need

My intention is not to write a general guide, but rather to warn you about two specific things you should be doing with analytics (Google Analytics in particular) as a result of changes Google is making because of GDPR.

Unexpected consequences of GDPR

When you deal directly with a person in the EU, and they give you personally identifiable information (PII) about themselves, you are typically in what is called the “data controller” role. The GDPR also identifies another role, which it calls “data processor,” which is any other company your company uses as a supplier and which handles that PII. When you use a product like Google Analytics on your website, Google is taking the role of data processor. While most of the restrictions of the GDPR apply to you as the controller, the processor must also comply, and it’s here that we see some potentially unintended (but possibly predictable) consequences of the legislation.

Google is unsurprisingly seeking to minimize their risk (I say it’s unsurprising because those GDPR fines could be as large as $4.4 billion based on last year’s revenue if they get it wrong). They are doing this firstly by pushing as much of the obligation onto you (the data controller) as possible, and secondly, by going further by default than the GDPR requires and being more aggressive than the regulation requires in shutting down accounts that infringe their terms (regardless of whether the infringement also infringes the GDPR).

This is entirely rational — with GA being in most cases a product offered for free, and the value coming to Google entirely in the aggregate, it makes perfect sense to limit their risks in ways that don’t degrade their value, and to just kick risky setups off the platform rather than taking on extreme financial risk for individual free accounts.

It’s not only Google, by the way. There are other suppliers doing similar things which will no doubt require similar actions, but I am focusing on Google here simply because GA is pervasive throughout the web marketing world. Some companies are even going as far as shutting down entirely for EU citizens (like unroll.me). See this Twitter thread of others.

Consequence 1: Default data retention settings for GA will delete your data

Starting on May 25, Google will be changing the default for data retention, meaning that if you don’t take action, certain data older than the cutoff will be automatically deleted.

You can read more about the details of the change on Krista Seiden’s personal blog (Krista works at Google, but this post is written in her personal capacity).

The reason I say that this isn’t strictly a GDPR thing is that it is related to changes Google is making on their end to ensure that they comply with their obligations as a data processor. It gives you tools you might need but isn’t strictly related to your GDPR compliance. There is no particular “right” answer to the question of how long you need to/should be/are allowed to keep this data stored in GA under the GDPR, but by my reading, given that it shouldn’t be PII anyway (see below) it isn’t really a GDPR question for most organizations. In particular, there is no particular reason to think that Google’s default is the correct/mandated/only setting you can choose under the GDPR.

Action: Review the promises being made by your legal team and your new privacy policy to understand the correct timeline setting for your org. In the absence of explicit promises to your users, my understanding is that you can retain any of this data you were allowed to capture in the first place unless you receive a deletion request against it. So while most orgs will have at least some changes to make to privacy policies at a minimum, most GA users can change back to retain this data indefinitely.

Consequence 2: Google is deleting GA accounts for capturing PII

It has long been against the Terms of Service to store any personally identifiable information (PII) in Google Analytics. Recently, though, it appears that Google has become far more diligent in checking for the presence of PII and robust in their handling of accounts found to contain any. Put more simply, Google will delete your account if they find PII.

It’s impossible to know for sure that this is GDPR-related, but being able if necessary to demonstrate to regulators that they are taking strict actions against anyone violating their PII-related terms is an obvious move for Google to reduce the risk they face as a Data Processor. It makes particular sense in an area where the vast majority of accounts are free accounts. Much like the previous point, and the reason I say that this is related to Google’s response to the GDPR coming into force, is that it would be perfectly possible to get your users’ permission to record their data in third-party services like GA, and fully comply with the regulations. Regardless of the permissions your users give you, Google’s GDPR-related crackdown (and heavier enforcement of the related terms that have been present for some time) means that it’s a new and greater risk than it was before.

Action: Audit your GA profile and implementation for PII risks:

• There are various ways you can search within GA itself to find data that could be personally identifying in places like page titles, URLs, custom data, etc. (see these two excellent guides)
• You can also audit your implementation by reviewing rules in tag manager and/or reviewing the code present on key pages. The most likely suspects are the places where people log in, take key actions on your site, give you additional personal information, or check out

Don’t take your EU law advice from big US tech companies

The internal effort and coordination required at Google to do their bit to comply even “just” as data processor is significant. Unfortunately, there are strong arguments that this kind of ostensibly user-friendly regulation which incurs outsize compliance burdens on smaller companies will cement the duopoly and dominance of Google and Facebook and enables them to pass the costs and burdens of compliance onto sectors that are already struggling.

Regardless of the intended or unintended consequences of the regulation, it seems clear to me that we shouldn’t be basing our own businesses’ (and our clients’) compliance on self-interested advice and actions from the tech giants. No matter how impressive their own compliance, I’ve been hugely underwhelmed by guidance content they’ve put out. See, for example, Google’s GDPR “checklist” — not exactly what I’d hope for:

So, while I’m not a lawyer, definitely not your lawyer, and this is not legal advice, if you haven’t already received any advice, I can say that you probably can’t just follow Google’s checklist to get compliant. But you should, as outlined above, take the specific actions you need to take to protect yourself and your business from their compliance activities.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by Angela_Petteys

If you’ve been on the Internet at all in the past few months, you’ve probably seen plenty of notices about privacy policy updates from one service or another. As a marketer, a few of those notices have most likely come from Google.

With the General Data Privacy Regulation (GDPR) set to go into effect on May 25th, 2018, many Internet services have been scrambling to get in compliance with the new standards — and Google is no exception. Given the nature of the services Google provides to marketers, GDPR absolutely made some significant changes in how they conduct business. And, in turn, some marketers may have to take steps to make sure their use of Google Analytics is allowable under the new rules. But a lot of marketers aren’t entirely sure what exactly GDPR is, what it means for their jobs, and what they need to do to follow the rules.

What is GDPR?

GDPR is a very broad reform that gives citizens who live in the European Economic Area (EEA) and Switzerland more control over how their personal data is collected and used online. GDPR introduces a lot of new rules and if you’re up for a little light reading, you can check out the full text of the regulation online. But here are a few of the most significant changes:

• Companies and other organizations have to be more transparent and clearly state what information they’re collecting, what it will be used for, how they’re collecting it, and if that information will be shared with anyone else. They can also only collect information that is directly relevant for its intended use. If the organization collecting that information later decides to use it for a different purpose, they must get permission again from each individual.
• GDPR also spells out how that information needs to be given to consumers. That information can no longer be hidden in long privacy policies filled with legal jargon. The information in disclosures needs to be written in plain language and “freely given, specific, informed, and unambiguous.” Individuals also have to take an action which clearly gives their consent to their information being collected. Pre-checked boxes and notices that rely on inaction as a way of giving consent will no longer be allowed. If a user does not agree to have their information collected, you cannot block them from accessing content based on that fact.
• Consumers also have the right to see what information a company has about them, request that incorrect information be corrected, revoke permission for their data to be saved, and have their data exported so they can switch to another service. If someone decides to revoke their permission, the organization needs to not only remove that information from their systems in a timely manner, they also need to have it removed from anywhere else they’ve shared that information.
• Organizations must also be able to give proof of the steps they’re taking to be in compliance. This can include keeping records of how people opt in to being on marketing lists and documentation regarding how customer information is being protected.
• Once an individual’s information has been collected, GDPR sets out requirements for how that information is stored and protected. If a data breach occurs, consumers must be notified within 72 hours. Failing to comply with GDPR can come with some very steep consequences. If a data breach occurs because of non-compliance, a company can be hit with fines as high as €20 million or 4% of the company’s annual global revenue, whichever amount is greater.

Do US-based businesses need to worry about GDPR?

Just because a business isn’t based in Europe doesn’t necessarily mean they’re off the hook as far as GDPR goes. If a company is based in the United States (or elsewhere outside the EEA), but conducts business in Europe, collects data about users from Europe, markets themselves in Europe, or has employees who work in Europe, GDPR applies to them, too.

Even if you’re working with a company that only conducts business in a very specific geographic area, you might occasionally get some visitors to your site from people outside of that region. For example, let’s say a pizza restaurant in Detroit publishes a blog post about the history of pizza on their site. It’s a pretty informative post and as a result, it brings in some traffic from pizza enthusiasts outside the Detroit area, including a few visitors from Spain. Would GDPR still apply in that sort of situation?

As long as it’s clear that a company’s goods or services are only available to consumers in the United States (or another country outside the EEA), GDPR does not apply. Going back to the pizza restaurant example, the other content on their site is written in English, emphasizes their Detroit location, and definitely doesn’t make any references to delivery to Spain, so those few page views from Spain wouldn’t be anything to worry about.

However, let’s say another US-based company has a site with the option to view German and French language versions of pages, lets customers pay with Euros, and uses marketing language that refers to European customers. In that situation, GDPR would apply since they are more clearly soliciting business from people in Europe.

Google Analytics & GDPR

If you use Google Analytics, Google is your data processor and since they handle data from people all over the world, they’ve had to take steps to become compliant with GDPR standards. However, you/your company are considered the data controller in this relationship and you will also need to take steps to make sure your Google Analytics account is set up to meet the new requirements.

Google has been rolling out some new features to help make this happen. In Analytics, you will now have the ability to delete the information of individual users if they request it. They’ve also introduced data retention settings which allow you to control how long individual user data is saved before being automatically deleted. Google has set this to be 26 months as the default setting, but if you are working with a US-based company that strictly conducts business in the United States, you can set it to never expire if you want to — at least until data protection laws change here, too. It’s important to note that this only applies to data about individual users and events, so aggregate data about high-level information like page views won’t be impacted by this.

To make sure you’re using Analytics in compliance with GDPR, a good place to start is by auditing all the data you collect to make sure it’s all relevant to its intended purpose and that you aren’t accidentally sending any personally identifiable information (PII) to Google Analytics. Sending PII to Google Analytics was already against its Terms of Service, but very often, it happens by accident when information is pushed through in a page URL. If it turns out you are sending PII to Analytics, you’ll need to talk to your web development team about how to fix it because using filters in Analytics to block it isn’t enough — you need to make sure it’s never sent to Google Analytics in the first place.

PII includes anything that can potentially be used to identify a specific person, either on its own or when combined with another piece of information, like an email address, a home address, a birthdate, a zip code, or an IP address. IP addresses weren’t always considered PII, but GDPR classifies them as an online identifier. Don’t worry, though — you can still get geographical insights about the visitors to your site. All you have to do is turn on IP anonymization and the last portion of an IP address will be replaced with a zero, so you can still get a general idea of where your traffic is coming from, although it will be a little less precise.

If you use Google Tag Manager, IP anonymization is pretty easy. Just open your Google Analytics tag or its settings variable, choose “More Settings,” and select “Fields to Set.” Then, choose “anonymizeip” in the “Field Name” box, enter “true” in the “Value” box,” and save your changes.

If you don’t use GTM, talk to your web development team about editing the Google Analytics code to anonymize IP addresses.

Pseudonymous information like user IDs and transaction IDs are still acceptable under GDPR, but it needs to be protected. User and transaction IDs need to be alphanumeric database identifiers, not written out in plain text.

Also, if you haven’t already done so, don’t forget to take the steps Google has mentioned in some of those emails they’ve sent out. If you’re based outside the EEA and GDPR applies to you, go into your Google Analytics account settings and accept the updated terms of processing. If you’re based in the EEA, the updated terms have already been included in your data processing terms. If GDPR applies to you, you’ll also need to go into your organization settings and provide contact information for your organization.

Privacy policies, forms, & cookie notices

Now that you’ve gone through your data and checked your settings in Google Analytics, you need to update your site’s privacy policy, forms, and cookie notices. If your company has a legal department, it may be best to involve them in this process to make sure you’re fully compliant.

Under GDPR, a site’s privacy policy needs to be clearly written in plain language and answer basic questions like what information is being collected, why it’s being collected, how it’s being collected, who is collecting it, how it will be used, and if it will be shared with anyone else. If your site is likely to be visited by children, this information needs to be written simply enough for a child to be able to understand it.

Forms and cookie notices also need to provide that kind of information. Cookie consent forms with really vague, generic messages like, “We use cookies to give you a better experience and by using this site, you agree to our policy,” are not GDPR compliant.

GDPR & other types of marketing

The impact GDPR will have on marketers isn’t just limited to how you use Google Analytics. If you use some particular types of marketing in the course of your job, you may have to make a few other changes, too.

Referral deals

If you work with a company that does “refer a friend”-type promotions where a customer has to enter information for a friend to receive a discount, GDPR is going to make a difference for you. Giving consent for data to be collected is a key part of GDPR and in these sorts of promotions, the person being referred can’t clearly consent to their information being collected. Under GDPR, it is possible to continue this practice, but it all depends on how that information is being used. If you store the information of the person being referred and use it for marketing purposes, it would be a violation of GDPR standards. However, if you don’t store that information or process it, you’re OK.

Email marketing

If you’re an email marketer and already follow best industry standards by doing things like only sending messages to those who clearly opt in to your list and making it easy for people to unsubscribe, the good news is that you’re probably in pretty good shape. As far as email marketing goes, GDPR is going to have the biggest impact on those who do things that have already been considered sketchy, like buying lists of contacts or not making it clear when someone is signing up to receive emails from you.

Even if you think you’re good to go, it’s still a good time to review your contacts and double check that your European contacts have indeed opted into being on your list and that it was clear what they were signing up for. If any of your contacts don’t have their country listed or you’re not sure how they opted in, you may want to either remove them from your list or put them on a separate segment so they don’t get any messages from you until you can get that figured out. Even if you’re confident your European contacts have opted in, there’s no harm in sending out an email asking them to confirm that they would like to continue receiving messages from you.

Creating a double opt-in process isn’t mandatory, but it would be a good idea since it helps remove any doubt over whether or not a person has agreed to being on your list. While you’re at it, take a look at the forms people use to sign up to be on your list and make sure they’re in line with GDPR standards, with no pre-checked boxes and the fact that they’re agreeing to receive emails from you is very clear.

For example, here’s a non-GDPR compliant email signup option I recently saw on a checkout page. They tell you what they’re planning to send to you, but the fact that it’s a pre-checked box placed underneath the more prominent “Place Order” button makes it very easy for people to unintentionally sign up for emails they might not actually want.

Jimmy Choo, on the other hand, also gives you the chance to sign up for emails while making a purchase, but since the box isn’t pre-checked, it’s good to go under GDPR.

Marketing automation

As is the case with standard email marketing, marketing automation specialists will need to make sure they have clear consent from everyone who has agreed to be part of their lists. Check your European contacts to make sure you know how they’ve opted in. Also review the ways people can opt into your list to make sure it’s clear what, exactly, they’re signing up for so that your existing contacts would be considered valid.

If you use marketing automation to re-engage customers who have been inactive for a while, you may need to get permission to contact them again, depending on how long it has been since they last interacted with you.

Some marketing automation platforms have functionality which will be impacted by GDPR. Lead scoring, for example, is now considered a form of profiling and you will need to get permission from individuals to have their information used in that way. Reverse IP tracking also needs consent.

It’s also important to make sure your marketing automation platform and CRM system are set to sync automatically. If a person on your list unsubscribes and continues receiving emails because of a lapse between the two, you could get in trouble for not being GDPR compliant.

Gated content

A lot of companies use gated content, like free reports, whitepapers, or webinars, as a way to generate leads. The way they see it, the person’s information serves as the price of admission. But since GDPR prohibits blocking access to content if a person doesn’t consent to their information being collected, is gated content effectively useless now?

GDPR doesn’t completely eliminate the possibility of gated content, but there are now higher standards for collecting user information. Basically, if you’re going to have gated content, you need to be able to prove that the information you collect is necessary for you to provide the deliverable. For example, if you were organizing a webinar, you’d be justified in collecting email addresses since attendees need to be sent a link to join in. You’d have a harder time claiming an email address was required for something like a whitepaper since that doesn’t necessarily have to be delivered via email. And of course, as with any other form on a site, forms for gated content need to clearly state all the necessary information about how the information being collected will be used.

If you don’t get a lot of leads from European users anyway, you may want to just block all gated content from European visitors. Another option would be to go ahead and make that information freely available to visitors from Europe.

Google AdWords

If you use Google AdWords to advertise to European residents, Google already required publishers and advertisers to get permission from end users by putting disclaimers on the landing page, but GDPR will be making some changes to these requirements. Google will now be requiring publishers to get clear consent from individuals to have their information collected. Not only does this mean you have to give more information about how a person’s information will be used, you’ll also need to keep records of consent and tell users how they can opt out later on if they want to do so. If a person doesn’t give consent to having their information collected, Google will be making it possible to serve them non-personalized ads.

In the end

GDPR is a significant change and trying to grasp the full scope of its changes is pretty daunting. This is far from being a comprehensive guide, so if you have any questions about how GDPR applies to a particular client you’re working with, it may be best to get in touch with their legal department or team. GDPR will impact some industries more than others, so it’s best to get some input from someone who truly understands the law and how it applies to that specific business.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by rjonesx.

We spend a lot of time discussing SEO tactics, but in a constantly changing industry, one thing that deserves more attention are the tactics agencies should employ in order to see success. From confidently raising your prices to knowing when to say no, Moz’s own Russ Jones covers four essential success tactics that’ll ultimately increase your bottom line in today’s edition of Whiteboard Friday.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Howdy, Moz fans. I am Russ Jones, and I can’t tell you how excited I am for my first Whiteboard Friday. I am Principal Search Scientist here at Moz. But before coming to Moz, for the 10 years prior to that, I was the Chief Technology Officer of a small SEO agency back in North Carolina. So I have a strong passion for agencies and consultants who are on the ground doing the work, helping websites rank better and helping build businesses.

So what I wanted to do today was spend a little bit of time talking about the lessons that I learned at an agency that admittedly I only learned through trial and error. But before we even go further, I just wanted to thank the folks at Hive Digital who I learned so much from, Jeff and Jake and Malcolm and Ryan, because the team effort over time is what ended up building an agency. Any agency that succeeds knows that that’s part of it. So we’ll start with that thank-you.

But what I really want to get into is that we spend a lot of time talking about SEO tactics, but not really about how to succeed in an industry that changes rapidly, in which there’s almost no certification, and where it can be difficult to explain to customers exactly how they’re going to be successful with what you offer. So what I’m going to do is break down four really important rules that I learned over the course of that 10 years. We’re going to go through each one of them as quickly as possible, but at the same time, hopefully you’ll walk away with some good ideas. Some of these are ones that it might at first feel a little bit awkward, but just follow me.

1. Raise prices

The first rule, number one in Let’s Make Money is raise your prices. Now, I remember quite clearly two years in to my job at Hive Digital — it was called Virante then — and we were talking about raising prices. We were just looking at our customers, saying to ourselves, “There’s no way they can afford it.” But then luckily we had the foresight that there was more to raising prices than just charging your customers more.

How it benefits old customers

The first thing that just hit us automatically was… “Well, with our old customers, we can just discount them. It’s not that bad. We’re in the same place as we always were.” But then it occurred to us, “Wait, wait, wait. If we discount our customers, then we’re actually increasing our perceived value.” Our existing customers now think, “Hey, they’re actually selling something better that’s more expensive, but I’m getting a deal,” and by offering them that deal because of their loyalty, you engender more loyalty. So it can actually be good for old customers.

How it benefits new customers

Now, for new customers, once again, same sort of situation. You’ve increased the perceived value. So your customers who come to you think, “Oh, this company is professional. This company is willing to invest. This company is interested in providing the highest quality of services.” In reality, because you’ve raised prices, you can. You can spend more time and money on each customer and actually do a better job. The third part is, “What’s the worst that could happen?” If they say no, you offer them the discount. You’re back where you started. You’re in the same position that you were before.

How it benefits your workers

Now, here’s where it really matters — your employees, your workers. If you are offering bottom line prices, you can’t offer them raises, you can’t offer them training, you can’t hire them help, or you can’t get better workers. But if you do, if you raise prices, the whole ecosystem that is your agency will do better.

How it improves your resources

Finally, and most importantly, which we’ll talk a little bit more later, is that you can finally tool up. You can get the resources and capital that you need to actually succeed. I drew this kind of out.

If we have a graph of quality of services that you offer and the price that you sell at, most agencies think that they’re offering great quality at a little price, but the reality is you’re probably down here. You’re probably under-selling your services and, because of that, you can’t offer the best that you can.

You should be up here. You should be offering higher quality, your experts who spend time all day studying this, and raising prices allows you to do that.

2. Schedule

Now, raising prices is only part one. The second thing is discipline, and I am really horrible about this. The reality is that I’m the kind of guy who looks for the latest and greatest and just jumps into it, but schedule matters. As hard as it is to admit it, I learned this from the CPC folks because they know that they have to stay on top of it every day of the week.

Well, here’s something that we kind of came up with as I was leaving the company, and that was to set all of our customers as much as possible into a schedule.

• Annually: we would handle keywords and competitors doing complete analysis.
• Semi-annually: Twice a year, we would do content analysis. What should you be writing about? What’s changed in your industry? What are different keywords that you might be able to target now given additional resources?
• Quarterly: You need to be looking at links. It’s just a big enough issue that you’ve got to look at it every couple of months, a complete link analysis.
• Monthly: You should be looking at your crawls. Moz will do that every week for you, but you should give your customers an idea, over the course of a month, what’s changed.
• Weekly: You should be doing rankings

But there are three things that, when you do all of these types of analysis, you need to keep in mind. Each one of them is a…

• Report
• Hours for consulting
• Phone call

This might seem like a little bit of overkill. But of course, if one of these comes back and nothing changed, you don’t need to do the phone call, but each one of these represents additional money in your pocket and importantly better service for your customers.

It might seem hard to believe that when you go to a customer and you tell them, “Look, nothing’s changed,” that you’re actually giving them value, but the truth is that if you go to the dentist and he tells you, you don’t have a cavity, that’s good news. You shouldn’t say to yourself at the end of the day, “Why’d I go to the dentist in the first place?” You should say, “I’m so glad I went to the dentist.” By that same positive outlook, you should be selling to your customers over and over and over again, hoping to give them the clarity they need to succeed.

3. Tool up!

So number three, you’re going to see this a lot in my videos because I just love SEO tools, but you’ve got to tool up. Once you’ve raised prices and you’re making more money with your customers, you actually can. Tools are superpowers. Tools allow you to do things that humans just can’t do. Like I can’t figure out the link graph on my own. I need tools to do it. But tools can do so much more than just auditing existing clients. For example, they can give you…

Better leads:

You can use tools to find opportunities.Take for example the tools within Moz and you want to find other car dealerships in the area that are really good and have an opportunity to rank, but aren’t doing as well as they should be in SERPs. You want to do this because you’ve already serviced successfully a different car dealership. Well, tools like Moz can do that. You don’t just have to use Moz to help your clients. You can use them to help yourself.

Better pre-audits:

Nobody walks into a sales call blind. You know who the website is. So you just start with a great pre-audit.

Faster workflows:

Which means you make more money quicker. If you can do your keyword analysis annually in half the time because you have the right tool for it, then you’re going to make far more money and be able to serve more customers.

Bulk pricing:

This one is just mind-blowingly simple. It’s bulk pricing. Every tool out there, the more you buy from them, the lower the price is. I remember at my old company sitting down at one point and recognizing that every customer that came in the door would need to spend about $1,000 on individual accounts to match what they were getting through us by being able to take advantage of the bulk discounts that we were getting as an agency by buying these seats on behalf of all of our customers.

So tell your clients when you’re talking to them on the phone, in the pitch be like, “Look, we use Moz, Majestic, Ahrefs, SEMrush,” list off all of the competitors. “We do Screaming Frog.” Just name them all and say, “If you wanted to go out and just get the data yourself from these tools, it would cost you more than we’re actually charging you.” The tools can sell themselves. You are saving them money.

4. Just say NO

Now, the last section, real quickly, are the things you’ve just got to learn to say no to. One of them has a little nuance to it. There’s going to be some bite back in the comments, I’m pretty sure, but I want to be careful with it.

No month-to-month contracts

The first thing to say no to is month-to-month contracts.

If a customer comes to you and they say, “Look, we want to do SEO, but we want to be able to cancel every 30 days.” the reality is this. They’re not interested in investing in SEO. They’re interested in dabbling in SEO. They’re interested in experimenting with SEO. Well, that’s not going to succeed. It’s only going to take one competitor or two who actually invest in it to beat them out, and when they beat them out, you’re going to look bad and they’re going to cancel their account with you. So sit down with them and explain to them that it is a long-term strategy and it’s just not worth it to your company to bring on customers who aren’t interested in investing in SEO. Say it politely, but just turn it away.

Don’t turn anything away

Now, notice that my next thing is don’t turn anything away. So here’s something careful. Here’s the nuance. It’s really important to learn to fire clients who are bad for your business, where you’re losing money on them or they’re just impolite, but that doesn’t mean you have to turn them away. You just need to turn them in the right direction. That right direction might be tools themselves. You can say, “Look, you don’t really need our consulting hours. You should go use these tools.” Or you can turn them to other fledgling businesses, friends you have in the industry who might be struggling at this time.

I’ll tell you a quick example. We don’t have much time, but many, many years ago, we had a client that came to us. At our old company, we had a couple of rules about who we would work with. We chose not to work in the adult industry. But at the time, I had a friend in the industry. He lived outside of the United States, and he had fallen on hard times. He literally had his business taken away from him via a series of just really unscrupulous events. I picked up the phone and gave him a call. I didn’t turn away the customer. I turned them over to this individual.

That very next year, he had ended up landing a new job at the top of one of the largest gambling organizations in the world. Well, frankly, they weren’t on our list of people we couldn’t work with. We landed the largest contract in the history of our company at that time, and it set our company straight for an entire year. It was just because instead of turning away the client, we turned them to a different direction. So you’ve got to say no to turning away everybody. They are opportunities. They might not be your opportunity, but they’re someone’s.

No service creep

The last one is service creep. Oh, man, this one is hard. A customer comes up to you and they list off three things that you offer that they want, and then they say, “Oh, yeah, we need social media management.” Somebody else comes up to you, three things you want to offer, and they say, “Oh yeah, we need you to write content,” and that’s not something you do. You’ve just got to not do that. You’ve got to learn to shave off services that you can’t offer. Instead, turn them over to people who can do them and do them very well.

What you’re going to end up doing in your conversation, your sales pitch is, “Look, I’m going to be honest with you. We are great at some things, but this isn’t our cup of tea. We know someone who’s really great at it.” That honesty, that candidness is just going to give them such a better relationship with you, and it’s going to build a stronger relationship with those other specialty companies who are going to send business your way. So it’s really important to learn to say no to say no service creep.

Well, anyway, there’s a lot that we went over there. I hope it wasn’t too much too fast, but hopefully we can talk more about it in the comments. I look forward to seeing you there. Thanks.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by MiriamEllis

I’ve advocated the use of Google’s owner response review feature since it first rolled out in 2010. This vital vehicle defends brand reputation and revenue, offering companies a means of transforming dissatisfied consumers into satisfied ones, supporting retention so that less has to be spent on new customer acquisition. I consider review responses to be a core customer service responsibility. Yet, eight years into the existence of this feature, marketing forums are still filled with entry-level questions like:

• Should I respond to reviews?
• Should I respond to positive reviews?
• How should I respond to negative reviews?

Over the years, I’ve seen different local SEO consultants reply in differing degrees to these common threads, but as of May 11, 2018, both agencies and brands woke to a new day: the day on which Google announced it would be emailing notifications like this to consumers when a business responds to their reviews, prompting them to view the reply.

Surveys indicate that well over 50% of consumers already expect responses within days of reviewing a business. With Google’s rollout, we can assume that this number is about to rise.

Why is this noteworthy news? I’ll explain exactly that in this post, plus demo how Moz Local can be a significant help to owners and marketers in succeeding in this new environment.

When “extra” becomes “expected”

In the past, owner responses may have felt like something extra a business could do to improve management of its reputation. Perhaps a company you’re marketing has been making the effort to respond to negative reviews, at the very least, but you’ve let replying to positive reviews slide. Or maybe you respond to reviews when you can get around to it, with days or weeks transpiring between consumer feedback and brand reaction.

Google’s announcement is important for two key reasons:

1) It signals that Google is turning reviews into a truly interactive feature, in keeping with so much else they’ve rolled out to the Knowledge Panel in recent times. Like booking buttons and Google Questions & Answers, notifications of owner responses are Google’s latest step towards making Knowledge Panels transactional platforms instead of static data entities. Every new feature brings us that much closer to Google positioning itself between providers and patrons for as many transactional moments as possible.

2) It signals a major turning point in consumer expectations. In the past, reviewers have left responses from motives of “having their say,” whether that’s to praise a business, warn fellow consumers, or simply document their experiences.

Now, imagine a patron who writes a negative review of two different restaurants he dined at for Sunday lunch and dinner. On Monday, he opens his email to find a Google notification that Restaurant A has left an owner response sincerely apologizing and reasonably explaining why service was unusually slow that weekend, but that Restaurant B is meeting his complaint about a rude waiter with dead air.

“So, Restaurant A cares about me, and Restaurant B couldn’t care less,” the consumer is left to conclude, creating an emotional memory that could inform whether he’s ever willing to give either business a second chance in the future.

Just one experience of receiving an owner response notification will set the rules of the game from here on out, making all future businesses that fail to respond seem inaccessible, neglectful, and even uncaring. It’s the difference between reviewers narrating their experiences from random motives, and leaving feedback with the expectation of being heard and answered.

I will go so far as to predict that Google’s announcement ups the game for all review platforms, because it will make owner responses to consumer sentiment an expected, rather than extra, effort.

The burden is on brands

Because no intelligent business would believe it can succeed in modern commerce while appearing unreachable or unconcerned, Google’s announcement calls for a priority shift. For brands large and small, it may not be an easy one, but it should look something like this:

• Negative reviews are now direct cries for help to our business; we will respond with whatever help we can give within X number of hours or days upon receipt
• Positive reviews are now thank-you notes directly to our company; we will respond with gratitude within X number of hours or days upon receipt

Defining X is going to have to depend on the resources of your organization, but in an environment in which consumers expect your reply, the task of responding must now be moved from the back burner to a hotter spot on the stovetop. Statistics differ in past assessments of consumer expectations of response times:

• In 2016, GetFiveStars found that 15.6% of consumers expected a reply with 1–3 hours, and 68.3% expected a reply within 1–3 days of leaving a review.
• In 2017, RevLocal found that 52% of consumers expected responses within 7 days.
• Overall, 30% of survey respondents told BrightLocal in 2017 that owner responses were a factor they looked at in judging a business.

My own expectation is that all of these numbers will now rise as a result of Google’s new function, meaning that the safest bet will be the fastest possible response. If resources are limited, I recommend prioritizing negative sentiment, aiming for a reply within hours rather than days as the best hope of winning back a customer. “Thank yous” for positive sentiment can likely wait for a couple of days, if absolutely necessary.

It’s inspiring to know that a recent Location3 study found that brands which do a good job of responding to reviews saw an average conversion rate of 13.9%, versus lackluster responders whose conversion rate was 10.4%. Depending on what you sell, those 3.5 points could be financially significant! But it’s not always easy to be optimally responsive.

If your business is small, accelerating response times can feel like a burden because of lack of people resources. If your business is a large, multi-location enterprise, the burden may lie in organizing awareness of hundreds of incoming reviews in a day, as well as keeping track of which reviews have been responded to and which haven’t.

The good news is…

Moz Local can help

The screenshot, above, is taken from the Moz Local dashboard. If you’re a customer, just log into your Moz Local account and go to your review section. From the “sources” section, choose “Google” — you’ll see the option to filter your reviews by “replied” and “not replied.” You’ll instantly be able to see which reviews you haven’t yet responded to. From there, simply use the in-dashboard feature that enables you to respond to your (or your clients’) reviews, without having to head over to your GMB dashboard or log into a variety of different clients’ GMB dashboards. So easy!

I highly recommend that all our awesome customers do this today and be sure you’ve responded to all of your most recent reviews. And, in the future, if you’re working your way through a stack of new, incoming Google reviews, this function should make it a great deal easier to keep organized about which ones you’ve checked off and which ones are still awaiting your response. I sincerely hope this function makes your work more efficient!

Need some help setting the right review response tone?

Please check out Mastering the Owner Response to the Quintet of Google My Business Reviews, which I published in 2016 to advocate responsiveness. It will walk you through these typical types of reviews you’ll be receiving:

• “I love you!”
• “I haven’t made up my mind yet.”
• “There was hair in my taco…”
• “I’m actually your competitor!”
• “I’m citing illegal stuff.”

The one update I’d make to the advice in the above piece, given Google’s rollout of the new notification function, would be to increase the number of positive reviews to which you’re responding. In 2016, I suggested that enterprises managing hundreds of locations should aim to express gratitude for at least 10% of favorable reviews. In 2018, I’d say reply with thanks to as many of these as you possibly can. Why? Because reviews are now becoming more transactional than ever, and if a customer says, “I like you,” it’s only polite to say, “Thanks!”. As more customers begin to expect responsiveness, failure to acknowledge praise could feel uncaring.

I would also suggest that responses to negative reviews more strongly feature a plea to the customer to contact the business so that things can be “made right.” GetFiveStars co-founder, Mike Blumenthal, is hoping that Google might one day create a private channel for brands and consumers to resolve complaints, but until that happens, definitely keep in mind that:

1. The new email alerts will ensure that more customers realize you’ve responded to their negative sentiment.
2. If, while “making things right” in the public response, you also urge the unhappy customer to let you make things “more right” in person, you will enhance your chances of retaining him.
3. If you are able to publicly or privately resolve a complaint, the customer may feel inspired to amend his review and raise your star rating; over time, more customers doing this could significantly improve your conversions and, possibly, your local search rankings.
4. All potential customers who see your active responses to complaints will understand that your policies are consumer-friendly, which should increase the likelihood of them choosing your business for transactions.

Looking ahead

One of the most interesting aspects I’m considering as of the rollout of response notifications is whether it may ultimately impact the tone of reviews themselves. In the past, some reviewers have given way to excesses in their sentiment, writing about companies in the ugliest possible language… language I’ve always wanted to hope they wouldn’t use face-to-face with other human beings at the place of business. I’m wondering now if knowing there’s a very good chance that brands responding to feedback could lessen the instances of consumers taking wild, often anonymous potshots at brands and create a more real-world, conversational environment.

In other words, instead of: “You overcharged me $3 for a soda and I know it’s because you’re [expletive] scammers, liars, crooks!!! Everyone beware of this company!!!”

We might see: “Hey guys, I just noticed a $3 overcharge on my receipt. I’m not too happy about this.”

The former scenario is honestly embarrassing. Trying to make someone feel better when they’ve just called you a thief feels a bit ridiculous and depressing. But the latter scenario is, at least, situation-appropriate instead of blown out of all proportion, creating an opening for you and your company to respond well and foster loyalty.

I can’t guarantee that reviewers will tone it down a bit if they feel more certain of being heard, but I’m hoping it will go that way in more and more cases.

What do you think? How will Google’s new function impact the businesses you market and the reviewers you serve? Please share your take and your tips with our community!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!