The Moz Hyperlinks API: An Introduction

What precisely IS an API? They’re these issues that you simply copy and paste lengthy unusual codes into Screaming Frog for hyperlinks information on a Website Crawl, proper?

I’m right here to let you know there’s a lot extra to them than that – in the event you’re keen to take just some little steps. However first, some fundamentals.

What’s an API?

API stands for “utility programming interface”, and it’s simply the way in which of… utilizing a factor. Every thing has an API. The net is a huge API that takes URLs as enter and returns pages.

However particular information companies just like the Moz Hyperlinks API have their very own algorithm. These guidelines range from service to service and could be a main stumbling block for individuals taking the following step.

When Screaming Frog offers you the additional hyperlinks columns in a crawl, it’s utilizing the Moz Hyperlinks API, however you may have this functionality wherever. For instance, all that tedious guide stuff you do in spreadsheet environments will be automated from data-pull to formatting and emailing a report.

If you happen to take this subsequent step, you will be extra environment friendly than your opponents, designing and delivering your individual search engine optimisation companies as a substitute of relying upon, paying for, and being restricted by the following proprietary product integration.

GET vs. POST

Most APIs you’ll encounter use the identical information transport mechanism as the online. Meaning there’s a URL concerned identical to a web site. Don’t get scared! It’s simpler than you assume. In some ways, utilizing an API is rather like utilizing a web site.

As with loading internet pages, the request could also be in considered one of two locations: the URL itself, or within the physique of the request. The URL is known as the “endpoint” and the usually invisibly submitted further a part of the request is known as the “payload” or “information”. When the information is within the URL, it’s referred to as a “question string” and signifies the “GET” methodology is used. You see this on a regular basis while you search:

https://www.google.com/search?q=moz+hyperlinks+api <-- GET methodology 

When the information of the request is hidden, it’s referred to as a “POST” request. You see this while you submit a type on the net and the submitted information doesn’t present on the URL. While you hit the again button after such a POST, browsers often warn you towards double-submits. The explanation the POST methodology is usually used is that you would be able to match much more within the request utilizing the POST methodology than the GET methodology. URLs would get very lengthy in any other case. The Moz Hyperlinks API makes use of the POST methodology.

Making requests

An online browser is what historically makes requests of internet sites for internet pages. The browser is a sort of software program generally known as a shopper. Purchasers are what make requests of companies. Extra than simply browsers could make requests. The flexibility to make shopper internet requests is usually constructed into programming languages like Python, or will be damaged out as a standalone device. The most well-liked instruments for making requests exterior a browser are curl and wget.

We’re discussing Python right here. Python has a built-in library referred to as URLLIB, nevertheless it’s designed to deal with so many several types of requests that it’s a little bit of a ache to make use of. There are different libraries which are extra specialised for making requests of APIs. The most well-liked for Python is known as requests. It’s so standard that it’s used for nearly each Python API tutorial you’ll discover on the net. So I’ll use it too. That is what “hitting” the Moz Hyperlinks API appears like:

response = requests.publish(endpoint, information=json_string, auth=auth_tuple)

Provided that every part was arrange appropriately (extra on that quickly), it will produce the next output:

{'next_token': 'JYkQVg4s9ak8iRBWDiz1qTyguYswnj035nqrQ1oIbW96IGJsb2dZgGzDeAM7Rw==',
 'outcomes': [{'anchor_text': 'moz',
              'external_pages': 7162,
              'external_root_domains': 2026}]}

That is JSON information. It is contained inside the response object that was returned from the API. It’s not on the drive or in a file. It’s in reminiscence. As long as it’s in reminiscence, you are able to do stuff with it (usually simply saving it to a file).

If you happen to needed to seize a bit of knowledge inside such a response, you might discuss with it like this:

response['results'][0]['external_pages']

This says: “Give me the primary merchandise within the outcomes record, after which give me the external_pages worth from that merchandise.” The end result can be 7162.

NOTE: If you happen to’re really following alongside executing code, the above line received’t work alone. There’s a specific amount of setup we’ll do shortly, together with putting in the requests library and organising a number of variables. However that is the essential thought.

JSON

JSON stands for JavaScript Object Notation. It’s a means of representing information in a means that’s simple for people to learn and write. It’s additionally simple for computer systems to learn and write. It’s a quite common information format for APIs that has considerably taken over the world because the older methods have been too tough for most individuals to make use of. Some individuals may name this a part of the “restful” API motion, however the rather more tough XML format can also be thought-about “restful” and everybody appears to have their very own interpretation. Consequently, I discover it finest to simply deal with JSON and the way it will get out and in of Python.

Python dictionaries

I lied to you. I stated that the information construction you have been above was JSON. Technically it’s actually a Python dictionary or dict datatype object. It’s a particular form of object in Python that’s designed to carry key/worth pairs. The keys are strings and the values will be any sort of object. The keys are just like the column names in a spreadsheet. The values are just like the cells within the spreadsheet. On this means, you may consider a Python dict as a JSON object. For instance right here’s making a dict in Python:

my_dict = {
    "identify": "Mike",
    "age": 52,
    "metropolis": "New York"
}

And right here is the equal in JavaScript:

var my_json = {
    "identify": "Mike",
    "age": 52,
    "metropolis": "New York"
}

Just about the identical factor, proper? Look intently. Key-names and string values get double-quotes. Numbers don’t. These guidelines apply constantly between JSON and Python dicts. In order you may think, it’s simple for JSON information to circulate out and in of Python. It is a nice present that has made fashionable API-work extremely accessible to the newbie by way of a device that has revolutionized the sector of knowledge science and is making inroads into advertising and marketing, Jupyter Notebooks.

Flattening information

However beware! As information flows between methods, it’s not unusual for the information to subtly change. For instance, the JSON information above may be transformed to a string. Strings may look precisely like JSON, however they’re not. They’re only a bunch of characters. Typically you’ll hear it referred to as “serializing”, or “flattening”. It’s a delicate level, however price understanding as it’s going to assist with one of many largest obstacles with the Moz Hyperlinks (and most JSON) APIs.

Objects have APIs

Precise JSON or dict objects have their very own little APIs for accessing the information within them. The flexibility to make use of these JSON and dict APIs goes away when the information is flattened right into a string, however it’s going to journey between methods extra simply, and when it arrives on the different finish, it is going to be “deserialized” and the API will come again on the opposite system.

Knowledge flowing between methods

That is the idea of transportable, interoperable information. Again when it was referred to as Digital Knowledge Interchange (or EDI), it was a really large deal. Then alongside got here the online after which XML after which JSON and now it’s only a regular a part of doing enterprise.

If you happen to’re in Python and also you wish to convert a dict to a flattened JSON string, you do the next:

import json

my_dict = {
    "identify": "Mike",
    "age": 52,
    "metropolis": "New York"
}

json_string = json.dumps(my_dict)

…which might produce the next output:

'{"identify": "Mike", "age": 52, "metropolis": "New York"}'

This appears virtually the identical as the unique dict, however in the event you look intently you may see that single-quotes are used across the complete factor. One other apparent distinction is that you would be able to line-wrap actual structured information for readability with none in poor health impact. You’ll be able to’t do it so simply with strings. That’s why it’s introduced all on one line within the above snippet.

Such stringifying processes are performed when passing information between completely different methods as a result of they aren’t at all times suitable. Regular textual content strings then again are suitable with virtually every part and will be handed on web-requests with ease. Such flattened strings of JSON information are incessantly known as the request.

Anatomy of a request

Once more, right here’s the instance request we made above:

response = requests.publish(endpoint, information=json_string, auth=auth_tuple)

Now that you simply perceive what the variable identify json_string is telling you about its contents, you shouldn’t be stunned to see that is how we populate that variable:

 data_dict = {
    "goal": "moz.com/weblog",
    "scope": "web page",
    "restrict": 1
}

json_string = json.dumps(data_dict)

…and the contents of json_string appears like this:

'{"goal": "moz.com/weblog", "scope": "web page", "restrict": 1}'

That is considered one of my key discoveries in studying the Moz Hyperlinks API. That is in frequent with numerous different APIs on the market however journeys me up each time as a result of it’s a lot extra handy to work with structured dicts than flattened strings. Nevertheless, most APIs anticipate the information to be a string for portability between methods, so we have now to transform it on the final second earlier than the precise API-call happens.

Pythonic hundreds and dumps

Now you might be questioning in that above instance, what a dump is doing in the midst of the code. The json.dumps() operate is known as a “dumper” as a result of it takes a Python object and dumps it right into a string. The json.hundreds() operate is known as a “loader” as a result of it takes a string and hundreds it right into a Python object.

The explanation for what look like singular and plural choices are literally binary and string choices. In case your information is binary, you utilize json.load() and json.dump(). In case your information is a string, you utilize json.hundreds() and json.dumps(). The s stands for string. Leaving the s off means binary.

Don’t let anyone let you know Python is ideal. It’s simply that its tough edges are usually not excessively objectionable.

Task vs. equality

For these of you fully new to Python or programming typically, what we’re doing once we hit the API is known as an task. The results of requests.publish() is being assigned to the variable named response.

response = requests.publish(endpoint, information=json_string, auth=auth_tuple)

We’re utilizing the = signal to assign the worth of the appropriate facet of the equation to the variable on the left facet of the equation. The variable response is now a reference to the thing that was returned from the API. Task is completely different from equality. The == signal is used for equality.

# That is task:
a = 1  # a is now equal to 1

# That is equality:
a == 1  # True, however depends that the above line has been executed

The POST methodology

response = requests.publish(endpoint, information=json_string, auth=auth_tuple)

The requests library has a operate referred to as publish() that takes 3 arguments. The primary argument is the URL of the endpoint. The second argument is the information to ship to the endpoint. The third argument is the authentication info to ship to the endpoint.

Key phrase parameters and their arguments

You could discover that a number of the arguments to the publish() operate have names. Names are set equal to values utilizing the = signal. Right here’s how Python capabilities get outlined. The primary argument is positional each as a result of it comes first and in addition as a result of there’s no key phrase. Keyworded arguments come after position-dependent arguments. Belief me, all of it is sensible after some time. All of us begin to assume like Guido van Rossum.

def arbitrary_function(argument1, identify=argument2):
    # do stuff

The identify within the above instance is known as a “key phrase” and the values that are available on these areas are referred to as “arguments”. Now arguments are assigned to variable names proper within the operate definition, so you may discuss with both argument1 or argument2 wherever inside this operate. If you happen to’d wish to be taught extra in regards to the guidelines of Python capabilities, you may examine them right here.

Organising the request

Okay, so let’s allow you to do every part crucial for that success assured second. We’ve been displaying the essential request:

response = requests.publish(endpoint, information=json_string, auth=auth_tuple)

…however we haven’t proven every part that goes into it. Let’s do this now. If you happen to’re following alongside and don’t have the requests library put in, you are able to do so with the next command from the identical terminal setting from which you run Python:

pip set up requests

Typically instances Jupyter can have the requests library put in already, however in case it doesn’t, you may set up it with the next command from inside a Pocket book cell:

!pip set up requests

And now we will put all of it collectively. There’s just a few issues right here which are new. A very powerful is how we’re taking 2 completely different variables and mixing them right into a single variable referred to as AUTH_TUPLE. You’ll have to get your individual ACCESSID and SECRETKEY from the Moz.com web site.

The API expects these two values to be handed as a Python information construction referred to as a tuple. A tuple is a listing of values that don’t change. I discover it fascinating that requests.publish() expects flattened strings for the information parameter, however expects a tuple for the auth parameter. I suppose it is sensible, however these are the delicate issues to grasp when working with APIs.

Right here’s the total code:

import json
import pprint
import requests

# Set Constants
ACCESSID = "mozscape-1234567890"  # Substitute along with your entry ID
SECRETKEY = "1234567890abcdef1234567890abcdef"  # Substitute along with your secret key
AUTH_TUPLE = (ACCESSID, SECRETKEY)

# Set Variables
endpoint = "https://lsapi.seomoz.com/v2/anchor_text"
data_dict = {"goal": "moz.com/weblog", "scope": "web page", "restrict": 1}
json_string = json.dumps(data_dict)

# Make the Request
response = requests.publish(endpoint, information=json_string, auth=AUTH_TUPLE)

# Print the Response
pprint(response.json())

…which outputs:

{'next_token': 'JYkQVg4s9ak8iRBWDiz1qTyguYswnj035nqrQ1oIbW96IGJsb2dZgGzDeAM7Rw==',
 'outcomes': [{'anchor_text': 'moz',
              'external_pages': 7162,
              'external_root_domains': 2026}]}

Utilizing all higher case for the AUTH_TUPLE variable is a conference many use in Python to point that the variable is a continuing. It’s not a requirement, nevertheless it’s a good suggestion to observe conventions when you may.

You could discover that I didn’t use all uppercase for the endpoint variable. That’s as a result of the anchor_text endpoint just isn’t a relentless. There are a selection of various endpoints that may take its place relying on what kind of lookup we needed to do. The alternatives are:

1. anchor_text

2. final_redirect

3. global_top_pages

4. global_top_root_domains

5. index_metadata

6. link_intersect

7. link_status

8. linking_root_domains

9. hyperlinks

10. top_pages

11. url_metrics

12. usage_data

And that leads into the Jupyter Pocket book that I ready on this subject positioned right here on Github. With this Pocket book you may prolong the instance I gave right here to any of the 12 accessible endpoints to create quite a lot of helpful deliverables, which would be the topic of articles to observe.