One place for hosting & domains

      How To Scrape Web Pages and Post Content to Twitter with Python 3


      The author selected The Computer History Museum to receive a donation as part of the Write for DOnations program.

      Introduction

      Twitter bots are a powerful way of managing your social media as well as extracting information from the microblogging network. By leveraging Twitter’s versatile APIs, a bot can do a lot of things: tweet, retweet, “favorite-tweet”, follow people with certain interests, reply automatically, and so on. Even though people can, and do, abuse their bot’s power, leading to a negative experience for other users, research shows that people view Twitter bots as a credible source of information. For example, a bot can keep your followers engaged with content even when you’re not online. Some bots even provide critical and helpful information, like @EarthquakesSF. The applications for bots are limitless. As of 2019, it is estimated that bots account for about 24% of all tweets on Twitter.

      In this tutorial, you’ll build a Twitter bot using this Twitter API library for Python. You’ll use API keys from your Twitter account to authorize your bot and build a to capable of scraping content from two websites. Furthermore, you’ll program your bot to alternately tweet content from these two websites and at set time intervals. Note that you’ll use Python 3 in this tutorial.

      Prerequisites

      You will need the following to complete this tutorial:

      Note: You’ll be setting up a developer account with Twitter, which involves an application review by Twitter before your can access the API keys you require for this bot. Step 1 walks through the specific details for completing the application.

      Step 1 — Setting Up Your Developer Account and Accessing Your Twitter API Keys

      Before you begin coding your bot, you’ll need the API keys for Twitter to recognize the requests of your bot. In this step, you’ll set up your Twitter Developer Account and access your API keys for your Twitter bot.

      To get your API keys, head over to developer.twitter.com and register your bot application with Twitter by clicking on Apply in the top right section of the page.

      Now click on Apply for a developer account.

      Next, click on Continue to associate your Twitter username with your bot application that you’ll be building in this tutorial.

      Twitter Username Association with Bot

      On the next page, for the purposes of this tutorial, you’ll choose the I am requesting access for my own personal use option since you’ll be building a bot for your own personal education use.

      Twitter API Personal Use

      After choosing your Account Name and Country, move on to the next section. For What use case(s) are you interested in?, pick the Publish and curate Tweets and Student project / Learning to code options. These categories are the best representation of why you’re completing this tutorial.

      Twitter Bot Purpose

      Then provide a description of the bot you’re trying to build. Twitter requires this to protect against bot abuse; in 2018 they introduced such vetting. For this tutorial, you’ll be scraping tech-focused content from The New Stack and The Coursera Blog.

      When deciding what to enter into the description box, model your answer on the following lines for the purposes of this tutorial:

      I’m following a tutorial to build a Twitter bot that will scrape content from websites like thenewstack.io (The New Stack) and blog.coursera.org (Coursera’s Blog) and tweet quotes from them. The scraped content will be aggregated and will be tweeted in a round-robin fashion via Python generator functions.

      Finally, choose no for Will your product, service, or analysis make Twitter content or derived information available to a government entity?

      Twitter Bot Intent

      Next, accept Twitter’s terms and conditions, click on Submit application, and then verify your email address. Twitter will send a verification email to you after your submission of this form.

      Once you verify your email, you’ll get an Application under review page with a feedback form for the application process.

      You will also receive another email from Twitter regarding the review:

      Application Review Email

      The timeline for Twitter’s application review process can vary significantly, but often Twitter will confirm this within a few minutes. However, should your application’s review take longer than this, it is not unusual, and you should receive it within a day or two. Once you receive confirmation, Twitter has authorized you to generate your keys. You can access these under the Keys and tokens tab after clicking the details button of your app on developer.twitter.com/apps.

      Finally go to the Permissions tab on your app’s page and set the Access Permission option to Read and Write since you want to write tweet content too. Usually, you would use the read-only mode for research purposes like analyzing trends, data-mining, and so on. The final option allows users to integrate chatbots into their existing apps, since chatbots require access to direct messages.

      Twitter App Permissions Page

      You have access to Twitter’s powerful API, which will be a crucial part of your bot application. Now you’ll set up your environment and begin building your bot.

      Step 2 — Building the Essentials

      In this step, you’ll write code to authenticate your bot with Twitter using the API keys, and make the first programmatic tweet via your Twitter handle. This will serve as a good milestone in your path towards the goal of building a Twitter bot that scrapes content from The New Stack and the Coursera Blog and tweets them periodically.

      First, you’ll set up a project folder and a specific programming environment for your project.

      Create your project folder:

      Move into your project folder:

      Then create a new Python virtual environment for your project:

      Then activate your environment using the following command:

      • source bird-env/bin/activate

      This will attach a (bird-env) prefix to the prompt in your terminal window.

      Now move to your text editor and create a file called credentials.py, which will store your Twitter API keys:

      Add the following content, replacing the highlighted code with your keys from Twitter:

      bird/credentials.py

      
      ACCESS_TOKEN='your-access-token'
      ACCESS_SECRET='your-access-secret'
      CONSUMER_KEY='your-consumer-key'
      CONSUMER_SECRET='your-consumer-secret'
      

      Now, you'll install the main API library for sending requests to Twitter. For this project, you'll require the following libraries: nltk, requests, twitter, lxml, random, and time. random and time are part of Python's standard library, so you don't need to separately install these libraries. To install the remaining libraries, you'll use pip, a package manager for Python.

      Open your terminal, ensure you're in the project folder, and run the following command:

      • pip3 install lxml nltk requests twitter
      • lxml and requests: You will use them for web scraping.
      • twitter: This is the library for making API calls to Twitter's servers.
      • nltk: (natural language toolkit) You will use to split paragraphs of blogs into sentences.
      • random: You will use this to randomly select parts of an entire scraped blog post.
      • time: You will use to make your bot sleep periodically after certain actions.

      Once you have installed the libraries, you're all set to begin programming. Now, you'll import your credentials into the main script that will run the bot. Alongside credentials.py, from your text editor create a file in the bird project directory, and name it bot.py:

      In practice, you would spread the functionality of your bot across multiple files as it grows more and more sophisticated. However, in this tutorial, you'll put all of your code in a single script, bot.py, for demonstration purposes.

      First you'll test your API keys by authorizing your bot. Begin by adding the following snippet to bot.py:

      bird/bot.py

      import random
      import time
      
      from lxml.html import fromstring
      import nltk
      nltk.download('punkt')
      import requests
      from twitter import OAuth, Twitter
      
      import credentials
      

      Here, you import the required libraries; and in a couple of instances you import the necessary functions from the libraries. You will use the fromstring function later in the code to convert the string source of a scraped webpage to a tree structure that makes it easier to extract relevant information from the page. OAuth will help you in constructing an authentication object from your keys, and Twitter will build the main API object for all further communication with Twitter's servers.

      Now extend bot.py with the following lines:

      bird/bot.py

      ...
      tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
      
      oauth = OAuth(
              credentials.ACCESS_TOKEN,
              credentials.ACCESS_SECRET,
              credentials.CONSUMER_KEY,
              credentials.CONSUMER_SECRET
          )
      t = Twitter(auth=oauth)
      

      nltk.download('punkt') downloads a dataset necessary for parsing paragraphs and tokenizing (splitting) them into smaller components. tokenizer is the object you'll use later in the code for splitting paragraphs written in English.

      oauth is the authentication object constructed by feeding the imported OAuth class with your API keys. You authenticate your bot via the line t = Twitter(auth=oauth). ACCESS_TOKEN and ACCESS_SECRET help in recognizing your application. Finally, CONSUMER_KEY and CONSUMER_SECRET help in recognizing the handle via which the application interacts with Twitter. You'll use this t object to communicate your requests to Twitter.

      Now save this file and run it in your terminal using the following command:

      Your output will look similar to the following, which means your authorization was successful:

      Output

      [nltk_data] Downloading package punkt to /Users/binaryboy/nltk_data... [nltk_data] Package punkt is already up-to-date!

      If you do receive an error, verify your saved API keys with those in your Twitter developer account and try again. Also ensure that the required libraries are installed correctly. If not, use pip3 again to install them.

      Now you can try tweeting something programmatically. Type the same command on the terminal with the -i flag to open the Python interpreter after the execution of your script:

      Next, type the following to send a tweet via your account:

      • t.statuses.update(status="Just setting up my Twttr bot")

      Now open your Twitter timeline in a browser, and you'll see a tweet at the top of your timeline containing the content you posted.

      First Programmatic Tweet

      Close the interpreter by typing quit() or CTRL + D.

      Your bot now has the fundamental capability to tweet. To develop your bot to tweet useful content, you'll incorporate web scraping in the next step.

      Step 3 — Scraping Websites for Your Tweet Content

      To introduce some more interesting content to your timeline, you'll scrape content from the New Stack and the Coursera Blog, and then post this content to Twitter in the form of tweets. Generally, to scrape the appropriate data from your target websites, you have to experiment with their HTML structure. Each tweet coming from the bot you'll build in this tutorial will have a link to a blog post from the chosen websites, along with a random quote from that blog. You'll implement this procedure within a function specific to scraping content from Coursera, so you'll name it scrape_coursera().

      First open bot.py:

      Add the scrape_coursera() function to the end of your file:

      bird/bot.py

      ...
      t = Twitter(auth=oauth)
      
      
      def scrape_coursera():
      

      To scrape information from the blog, you'll first request the relevant webpage from Coursera's servers. For that you will use the get() function from the requests library. get() takes in a URL and fetches the corresponding webpage. So, you'll pass blog.coursera.org as an argument to get(). But you also need to provide a header in your GET request, which will ensure Coursera's servers recognize you as a genuine client. Add the following highlighted lines to your scrape_coursera() function to provide a header:

      bird/bot.py

      def scrape_coursera():
          HEADERS = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                            ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
              }
      

      This header will contain information pertaining to a defined web browser running on a specific operating system. As long as this information (usually referred to as User-Agent) corresponds to real web browsers and operating systems, it doesn't matter whether the header information aligns with the actual web browser and operating system on your computer. Therefore this header will work fine for all systems.

      Once you have defined the headers, add the following highlighted lines to make a GET request to Coursera by specifying the URL of the blog webpage:

      bird/bot.py

      ...
      def scrape_coursera():
          HEADERS = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                            ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
              }
          r = requests.get('https://blog.coursera.org', headers=HEADERS)
          tree = fromstring(r.content)
      

      This will fetch the webpage to your machine and save the information from the entire webpage in the variable r. You can assess the HTML source code of the webpage using the content attribute of r. Therefore, the value of r.content is the same as what you see when you inspect the webpage in your browser by right clicking on the page and choosing the Inspect Element option.

      Here you've also added the fromstring function. You can pass the webpage's source code to the fromstring function imported from the lxml library to construct the tree structure of the webpage. This tree structure will allow you to conveniently access different parts of the webpage. HTML source code has a particular tree-like structure; every element is enclosed in the <html> tag and nested thereafter.

      Now, open https://blog.coursera.org in a browser and inspect its HTML source using the browser's developer tools. Right click on the page and choose the Inspect Element option. You'll see a window appear at the bottom of the browser, showing part of the page's HTML source code.

      browser-inspect

      Next, right click on the thumbnail of any visible blog post and then inspect it. The HTML source will highlight the relevant HTML lines where that blog thumbnail is defined. You'll notice that all blog posts on this page are defined within a <div> tag with a class of "recent":

      blog-div

      Thus, in your code, you'll use all such blog post div elements via their XPath, which is a convenient way of addressing elements of a web page.

      To do so, extend your function in bot.py as follows:

      bird/bot.py

      ...
      def scrape_coursera():
          HEADERS = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                            ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
                          }
          r = requests.get('https://blog.coursera.org', headers=HEADERS)
          tree = fromstring(r.content)
          links = tree.xpath('//div[@class="recent"]//div[@class="title"]/a/@href')
          print(links)
      
      scrape_coursera()
      

      Here, the XPath (the string passed to tree.xpath()) communicates that you want div elements from the entire web page source, of class "recent". The // corresponds to searching the whole webpage, div tells the function to extract only the div elements, and [@class="recent"] asks it to only extract those div elements that have the values of their class attribute as "recent".

      However, you don't need these elements themselves, you only need the links they're pointing to, so that you can access the individual blog posts to scrape their content. Therefore, you extract all the links using the values of the href anchor tags that are within the previous div tags of the blog posts.

      To test your program so far, you call the scrape_coursera() function at the end of bot.py.

      Save and exit bot.py.

      Now run bot.py with the following command:

      In your output, you'll see a list of URLs like the following:

      Output

      ['https://blog.coursera.org/career-stories-from-inside-coursera/', 'https://blog.coursera.org/unlock-the-power-of-data-with-python-university-of-michigan-offers-new-programming-specializations-on-coursera/', ...]

      After you verify the output, you can remove the last two highlighted lines from bot.py script:

      bird/bot.py

      ...
      def scrape_coursera():
          ...
          tree = fromstring(r.content)
          links = tree.xpath('//div[@class="recent"]//div[@class="title"]/a/@href')
          ~~print(links)~~
      
      ~~scrape_coursera()~~
      

      Now extend the function in bot.py with the following highlighted line to extract the content from a blog post:

      bird/bot.py

      ...
      def scrape_coursera():
          ...
          links = tree.xpath('//div[@class="recent"]//div[@class="title"]/a/@href')
          for link in links:
              r = requests.get(link, headers=HEADERS)
              blog_tree = fromstring(r.content)
      

      You iterate over each link, fetch the corresponding blog post, extract a random sentence from the post, and then tweet this sentence as a quote, along with the corresponding URL. Extracting a random sentence involves three parts:

      1. Grabbing all the paragraphs in the blog post as a list.
      2. Selecting a paragraph at random from the list of paragraphs.
      3. Selecting a sentence at random from this paragraph.

      You'll execute these steps for each blog post. For fetching one, you make a GET request for its link.

      Now that you have access to the content of a blog, you will introduce the code that executes these three steps to extract the content you want from it. Add the following extension to your scraping function that executes the three steps:

      bird/bot.py

      ...
      def scrape_coursera():
          ...
          for link in links:
              r = requests.get(link, headers=HEADERS)
              blog_tree = fromstring(r.content)
              paras = blog_tree.xpath('//div[@class="entry-content"]/p')
              paras_text = [para.text_content() for para in paras if para.text_content()]
              para = random.choice(paras_text)
              para_tokenized = tokenizer.tokenize(para)
              for _ in range(10):
                  text = random.choice(para)
                  if text and 60 < len(text) < 210:
                      break
      

      If you inspect the blog post by opening the first link, you'll notice that all the paragraphs belong to the div tag having entry-content as its class. Therefore, you extract all paragraphs as a list with paras = blog_tree.xpath('//div[@class="entry-content"]/p').

      Div Enclosing Paragraphs

      The list elements aren't literal paragraphs; they are Element objects. To extract the text out of these objects, you use the text_content() method. This line follows Python's list comprehension design pattern, which defines a collection using a loop that is usually written out in a single line. In bot.py, you extract the text for each paragraph element object and store it in a list if the text is not empty. To randomly choose a paragraph from this list of paragraphs, you incorporate the random module.

      Finally, you have to select a sentence at random from this paragraph, which is stored in the variable para. For this task, you first break the paragraph into sentences. One approach to accomplish this is using the Python's split() method. However this can be difficult since a sentence can be split at multiple breakpoints. Therefore, to simplify your splitting tasks, you leverage natural language processing through the nltk library. The tokenizer object you defined earlier in the tutorial will be useful for this purpose.

      Now that you have a list of sentences, you call random.choice() to extract a random sentence. You want this sentence to be a quote for a tweet, so it can't exceed 280 characters. However, for aesthetic reasons, you'll select a sentence that is neither too big nor too small. You designate that your tweet sentence should have a length between 60 to 210 characters. The sentence random.choice() picks might not satisfy this criterion. To identify the right sentence, your script will make ten attempts, checking for the criterion each time. Once the randomly picked-up sentence satisfies your criterion, you can break out of the loop.

      Although the probability is quite low, it is possible that none of the sentences meet this size condition within ten attempts. In this case, you'll ignore the corresponding blog post and move on to the next one.

      Now that you have a sentence to quote, you can tweet it with the corresponding link. You can do this by yielding a string that contains the randomly picked-up sentence as well as the corresponding blog link. The code that calls this scrape_coursera() function will then post the yielded string to Twitter via Twitter's API.

      Extend your function as follows:

      bird/bot.py

      ...
      def scrape_coursera():
          ...
          for link in links:
              ...
              para_tokenized = tokenizer.tokenize(para)
              for _ in range(10):
                  text = random.choice(para)
                  if text and 60 < len(text) < 210:
                      break
              else:
                  yield None
              yield '"%s" %s' % (text, link)
      

      The script only executes the else statement when the preceding for loop doesn't break. Thus, it only happens when the loop is not able to find a sentence that fits your size condition. In that case, you simply yield None so that the code that calls this function is able to determine that there is nothing to tweet. It will then move on to call the function again and get the content for the next blog link. But if the loop does break it means the function has found an appropriate sentence; the script will not execute the else statement, and the function will yield a string composed of the sentence as well as the blog link, separated by a single whitespace.

      The implementation of the scrape_coursera() function is almost complete. If you want to make a similar function to scrape another website, you will have to repeat some of the code you've written for scraping Coursera's blog. To avoid rewriting and duplicating parts of the code and to ensure your bot's script follows the DRY principle (Don't Repeat Yourself), you'll identify and abstract out parts of the code that you will use again and again for any scraper function written later.

      Regardless of the website the function is scraping, you'll have to randomly pick up a paragraph and then choose a random sentence from this chosen paragraph — you can extract out these functionalities in separate functions. Then you can simply call these functions from your scraper functions and achieve the desired result. You can also define HEADERS outside the scrape_coursera() function so that all of the scraper functions can use it. Therefore, in the code that follows, the HEADERS definition should precede that of the scraper function, so that eventually you're able to use it for other scrapers:

      bird/bot.py

      ...
      HEADERS = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                        ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
          }
      
      
      def scrape_coursera():
          r = requests.get('https://blog.coursera.org', headers=HEADERS)
          ...
      

      Now you can define the extract_paratext() function for extracting a random paragraph from a list of paragraph objects. The random paragraph will pass to the function as a paras argument, and return the chosen paragraph's tokenized form that you'll use later for sentence extraction:

      bird/bot.py

      ...
      HEADERS = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                            ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
              }
      
      def extract_paratext(paras):
          """Extracts text from <p> elements and returns a clean, tokenized random
          paragraph."""
      
          paras = [para.text_content() for para in paras if para.text_content()]
          para = random.choice(paras)
          return tokenizer.tokenize(para)
      
      
      def scrape_coursera():
          r = requests.get('https://blog.coursera.org', headers=HEADERS)
          ...
      

      Next, you will define a function that will extract a random sentence of suitable length (between 60 and 210 characters) from the tokenized paragraph it gets as an argument, which you can name as para. If such a sentence is not discovered after ten attempts, the function returns None instead. Add the following highlighted code to define the extract_text() function:

      bird/bot.py

      ...
      
      def extract_paratext(paras):
          ...
          return tokenizer.tokenize(para)
      
      
      def extract_text(para):
          """Returns a sufficiently-large random text from a tokenized paragraph,
          if such text exists. Otherwise, returns None."""
      
          for _ in range(10):
              text = random.choice(para)
              if text and 60 < len(text) < 210:
                  return text
      
          return None
      
      
      def scrape_coursera():
          r = requests.get('https://blog.coursera.org', headers=HEADERS)
          ...
      

      Once you have defined these new helper functions, you can redefine the scrape_coursera() function to look as follows:

      bird/bot.py

      ...
      def extract_paratext():
          for _ in range(10):<^>
              text = random.choice(para)
          ...
      
      
      def scrape_coursera():
          """Scrapes content from the Coursera blog."""
      
          url = 'https://blog.coursera.org'
          r = requests.get(url, headers=HEADERS)
          tree = fromstring(r.content)
          links = tree.xpath('//div[@class="recent"]//div[@class="title"]/a/@href')
      
          for link in links:
              r = requests.get(link, headers=HEADERS)
              blog_tree = fromstring(r.content)
              paras = blog_tree.xpath('//div[@class="entry-content"]/p')
              para = extract_paratext(paras)
              text = extract_text(para)
              if not text:
                  continue
      
              yield '"%s" %s' % (text, link)
      

      Save and exit bot.py.

      Here you're using yield instead of return because, for iterating over the links, the scraper function will give you the tweet strings one-by-one in a sequential fashion. This means when you make a first call to the scraper sc defined as sc = scrape_coursera(), you will get the tweet string corresponding to the first link among the list of links that you computed within the scraper function. If you run the following code in the interpreter, you'll get string_1 and string_2 as displayed below, if the links variable within scrape_coursera() holds a list that looks like ["https://thenewstack.io/cloud-native-live-twistlocks-virtual-conference/", "https://blog.coursera.org/unlock-the-power-of-data-with-python-university-of-michigan-offers-new-programming-specializations-on-coursera/", ...].

      Instantiate the scraper and call it sc:

      >>> sc = scrape_coursera()
      

      It is now a generator; it generates or scrapes relevant content from Coursera, one at a time. You can access the scraped content one-by-one by calling next() over sc sequentially:

      >>> string_1 = next(sc)
      >>> string_2 = next(sc)
      

      Now you can print the strings you've defined to display the scraped content:

      >>> print(string_1)
      "Other speakers include Priyanka Sharma, director of cloud native alliances at GitLab and Dan Kohn, executive director of the Cloud Native Computing Foundation." https://thenewstack.io/cloud-native-live-twistlocks-virtual-conference/
      >>>
      >>> print(string_2)
      "You can learn how to use the power of Python for data analysis with a series of courses covering fundamental theory and project-based learning." https://blog.coursera.org/unlock-the-power-of-data-with-python-university-of-michigan-offers-new-programming-specializations-on-coursera/
      >>>
      

      If you use return instead, you will not be able to obtain the strings one-by-one and in a sequence. If you simply replace the yield with return in scrape_coursera(), you'll always get the string corresponding to the first blog post, instead of getting the first one in the first call, second one in the second call, and so on. You can modify the function to simply return a list of all the strings corresponding to all the links, but that is more memory intensive. Also, this kind of program could potentially make a lot of requests to Coursera's servers within a short span of time if you want the entire list quickly. This could result in your bot getting temporarily banned from accessing a website. Therefore, yield is the best fit for a wide variety of scraping jobs, where you only need information scraped one-at-a-time.

      Step 4 — Scraping Additional Content

      In this step, you'll build a scraper for thenewstack.io. The process is similar to what you've completed in the previous step, so this will be a quick overview.

      Open the website in your browser and inspect the page source. You'll find here that all blog sections are div elements of class normalstory-box.

      HTML Source Inspection of The New Stack website

      Now you'll make a new scraper function named scrape_thenewstack() and make a GET request to thenewstack.io from within it. Next, extract the links to the blogs from these elements and then iterate over each link. Add the following code to achieve this:

      bird/bot.py

      ...
      def scrape_coursera():
          ...
          yield '"%s" %s' % (text, link)
      
      
      def scrape_thenewstack():
          """Scrapes news from thenewstack.io"""
      
          r = requests.get('https://thenewstack.io', verify=False)
      
              tree = fromstring(r.content)
              links = tree.xpath('//div[@class="normalstory-box"]/header/h2/a/@href')
              for link in links:
      

      You use the verify=False flag because websites can sometimes have expired security certificates and it's OK to access them if no sensitive data is involved, as is the case here. The verify=False flag tells the requests.get method to not verify the certificates and continue fetching data as usual. Otherwise, the method throws an error about expired security certificates.

      You can now extract the paragraphs of the blog corresponding to each link, and use the extract_paratext() function you built in the previous step to pull out a random paragraph from the list of available paragraphs. Finally, extract a random sentence from this paragraph using the extract_text() function, and then yield it with the corresponding blog link. Add the following highlighted code to your file to accomplish these tasks:

      bird/bot.py

      ...
      def scrape_thenewstack():
          ...
          links = tree.xpath('//div[@class="normalstory-box"]/header/h2/a/@href')
      
          for link in links:
              r = requests.get(link, verify=False)
              tree = fromstring(r.content)
              paras = tree.xpath('//div[@class="post-content"]/p')
              para = extract_paratext(paras)
              text = extract_text(para)  
              if not text:
                  continue
      
              yield '"%s" %s' % (text, link)
      

      You now have an idea of what a scraping process generally encompasses. You can now build your own, custom scrapers that can, for example, scrape the images in blog posts instead of random quotes. For that, you can look for the relevant <img> tags. Once you have the right path for tags, which serve as their identifiers, you can access the information within tags using the names of corresponding attributes. For example, in the case of scraping images, you can access the links of images using their src attributes.

      At this point, you've built two scraper functions for scraping content from two different websites, and you've also built two helper functions to reuse functionalities that are common across the two scrapers. Now that your bot knows how to tweet and what to tweet, you'll write the code to tweet the scraped content.

      Step 5 — Tweeting the Scraped Content

      In this step, you'll extend the bot to scrape content from the two websites and tweet it via your Twitter account. More precisely, you want it to tweet content from the two websites alternately, and at regular intervals of ten minutes, for an indefinite period of time. Thus, you will use an infinite while loop to implement the desired functionality. You'll do this as part of a main() function, which will implement the core high-level process that you'll want your bot to follow:

      bird/bot.py

      ...
      def scrape_thenewstack():
          ...
          yield '"%s" %s' % (text, link)
      
      
      def main():
          """Encompasses the main loop of the bot."""
          print('---Bot started---n')
          news_funcs = ['scrape_coursera', 'scrape_thenewstack']
          news_iterators = []  
          for func in news_funcs:
              news_iterators.append(globals()[func]())
          while True:
              for i, iterator in enumerate(news_iterators):
                  try:
                      tweet = next(iterator)
                      t.statuses.update(status=tweet)
                      print(tweet, end='nn')
                      time.sleep(600)  
                  except StopIteration:
                      news_iterators[i] = globals()[newsfuncs[i]]()
      

      You first create a list of the names of the scraping functions you defined earlier, and call it as news_funcs. Then you create an empty list that will hold the actual scraper functions, and name that list as news_iterators. You then populate it by going through each name in the news_funcs list and appending the corresponding iterator in the news_iterators list. You're using Python's built-in globals() function. This returns a dictionary that maps variable names to actual variables within your script. An iterator is what you get when you call a scraper function: for example, if you write coursera_iterator = scrape_coursera(), then coursera_iterator will be an iterator on which you can invoke next() calls. Each next() call will return a string containing a quote and its corresponding link, exactly as defined in the scrape_coursera() function's yield statement. Each next() call goes through one iteration of the for loop in the scrape_coursera() function. Thus, you can only make as many next() calls as there are blog links in the scrape_coursera() function. Once that number exceeds, a StopIteration exception will be raised.

      Once both the iterators populate the news_iterators list, the main while loop starts. Within it, you have a for loop that goes through each iterator and tries to obtain the content to be tweeted. After obtaining the content, your bot tweets it and then sleeps for ten minutes. If the iterator has no more content to offer, a StopIteration exception is raised, upon which you refresh that iterator by re-instantiating it, to check for the availability of newer content on the source website. Then you move on to the next iterator, if available. Otherwise, if execution reaches the end of the iterators list, you restart from the beginning and tweet the next available content. This makes your bot tweet content alternately from the two scrapers for as long as you want.

      All that remains now is to make a call to the main() function. You do this when the script is called directly by the Python interpreter:

      bird/bot.py

      ...
      def main():
          print('---Bot started---n')<^>
          news_funcs = ['scrape_coursera', 'scrape_thenewstack']
          ...
      
      if __name__ == "__main__":  
          main()
      

      The following is a completed version of the bot.py script. You can also view the script on this GitHub repository.

      bird/bot.py

      
      """Main bot script - bot.py
      For the DigitalOcean Tutorial.
      """
      
      
      import random
      import time
      
      
      from lxml.html import fromstring
      import nltk  
      nltk.download('punkt')
      import requests  
      
      from twitter import OAuth, Twitter
      
      
      import credentials
      
      tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
      
      oauth = OAuth(
              credentials.ACCESS_TOKEN,
              credentials.ACCESS_SECRET,
              credentials.CONSUMER_KEY,
              credentials.CONSUMER_SECRET
          )
      t = Twitter(auth=oauth)
      
      HEADERS = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'
                            ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'
              }
      
      
      def extract_paratext(paras):
          """Extracts text from <p> elements and returns a clean, tokenized random
          paragraph."""
      
          paras = [para.text_content() for para in paras if para.text_content()]
          para = random.choice(paras)
          return tokenizer.tokenize(para)
      
      
      def extract_text(para):
          """Returns a sufficiently-large random text from a tokenized paragraph,
          if such text exists. Otherwise, returns None."""
      
          for _ in range(10):
              text = random.choice(para)
              if text and 60 < len(text) < 210:
                  return text
      
          return None
      
      
      def scrape_coursera():
          """Scrapes content from the Coursera blog."""
          url = 'https://blog.coursera.org'
          r = requests.get(url, headers=HEADERS)
          tree = fromstring(r.content)
          links = tree.xpath('//div[@class="recent"]//div[@class="title"]/a/@href')
      
          for link in links:
              r = requests.get(link, headers=HEADERS)
              blog_tree = fromstring(r.content)
              paras = blog_tree.xpath('//div[@class="entry-content"]/p')
              para = extract_paratext(paras)  
              text = extract_text(para)  
              if not text:
                  continue
      
              yield '"%s" %s' % (text, link)  
      
      
      def scrape_thenewstack():
          """Scrapes news from thenewstack.io"""
      
          r = requests.get('https://thenewstack.io', verify=False)
      
          tree = fromstring(r.content)
          links = tree.xpath('//div[@class="normalstory-box"]/header/h2/a/@href')
      
          for link in links:
              r = requests.get(link, verify=False)
              tree = fromstring(r.content)
              paras = tree.xpath('//div[@class="post-content"]/p')
              para = extract_paratext(paras)
              text = extract_text(para)  
              if not text:
                  continue
      
              yield '"%s" %s' % (text, link)
      
      
      def main():
          """Encompasses the main loop of the bot."""
          print('Bot started.')
          news_funcs = ['scrape_coursera', 'scrape_thenewstack']
          news_iterators = []  
          for func in news_funcs:
              news_iterators.append(globals()[func]())
          while True:
              for i, iterator in enumerate(news_iterators):
                  try:
                      tweet = next(iterator)
                      t.statuses.update(status=tweet)
                      print(tweet, end='n')
                      time.sleep(600)
                  except StopIteration:
                      news_iterators[i] = globals()[newsfuncs[i]]()
      
      
      if __name__ == "__main__":  
          main()
      
      

      Save and exit bot.py.

      The following is a sample execution of bot.py:

      You will receive output showing the content that your bot has scraped, in a similar format to the following:

      Output

      [nltk_data] Downloading package punkt to /Users/binaryboy/nltk_data... [nltk_data] Package punkt is already up-to-date! ---Bot started--- "Take the first step toward your career goals by building new skills." https://blog.coursera.org/career-stories-from-inside-coursera/ "Other speakers include Priyanka Sharma, director of cloud native alliances at GitLab and Dan Kohn, executive director of the Cloud Native Computing Foundation." https://thenewstack.io/cloud-native-live-twistlocks-virtual-conference/ "You can learn how to use the power of Python for data analysis with a series of courses covering fundamental theory and project-based learning." https://blog.coursera.org/unlock-the-power-of-data-with-python-university-of-michigan-offers-new-programming-specializations-on-coursera/ "“Real-user monitoring is really about trying to understand the underlying reasons, so you know, ‘who do I actually want to fly with?" https://thenewstack.io/how-raygun-co-founder-and-ceo-spun-gold-out-of-monitoring-agony/

      After a sample run of your bot, you'll see a full timeline of programmatic tweets posted by your bot on your Twitter page. It will look something like the following:

      Programmatic Tweets posted

      As you can see, the bot is tweeting the scraped blog links with random quotes from each blog as highlights. This feed is now an information feed with tweets alternating between blog quotes from Coursera and thenewstack.io. You've built a bot that aggregates content from the web and posts it on Twitter. You can now broaden the scope of this bot as per your wish by adding more scrapers for different websites, and the bot will tweet content coming from all the scrapers in a round-robin fashion, and in your desired time intervals.

      Conclusion

      In this tutorial you built a basic Twitter bot with Python and scraped some content from the web for your bot to tweet. There are many bot ideas to try; you could also implement your own ideas for a bot's utility. You can combine the versatile functionalities offered by Twitter's API and create something more complex. For a version of a more sophisticated Twitter bot, check out chirps, a Twitter bot framework that uses some advanced concepts like multithreading to make the bot do multiple things simultaneously. There are also some fun-idea bots, like misheardly. There are no limits on the creativity one can use while building Twitter bots. Finding the right API endpoints to hit for your bot's implementation is essential.

      Finally, bot etiquette or ("botiquette") is important to keep in mind when building your next bot. For example, if your bot incorporates retweeting, make all tweets' text pass through a filter to detect abusive language before retweeting them. You can implement such features using regular expressions and natural language processing. Also, while looking for sources to scrape, follow your judgment and avoid ones that spread misinformation. To read more about botiquette, you can visit this blog post by Joe Mayo on the topic.





      Source link

      Stop Trying to Get Lucky — Here’s How to Design Landing Pages That Convert


      Don’t be so humble: you’ve spent a lot of time developing your website’s marketing strategy, right?

      Between developing social media posts (and outsmarting that tricky Facebook algorithm), choosing quality web hosting and branding your website, and learning the ins-and-outs of SEO, you’ve spent countless hours — and probably a pretty penny — getting internet crowds to notice your content.

      For that, you deserve a pat on the back and a big gold star. Hooray for you!

      But here’s a kicker: have you thought about what visitors experience when they click your ads and visit your site? You’ve got people heading towards your page, but what do they do when they get there? You’re acing the marketing push to drive traffic to your site, but . . . then what?

      If they’re not converting — that is, if you’re not generating leads — then red flag! There’s a problem. If conversions seem out of reach, what do you do? Rub your lucky rabbit’s foot? Wish on a shooting star?

      Nope, you don’t need to go hunting for four-leaf clovers to generate leads. Getting site visitors to convert isn’t a matter of luck — it’s about killer landing pages. Armed with the right knowledge and tools, you website owners can attract more traffic to your site — and convert visitors — with top-of-the-line landing pages.

      Our comprehensive (we mean it!) guide covers all the need-to-know info, like the key elements of a strong landing page, best practices for copy and CTAs, and more. Plus, we’ve loaded it chock-full of real-life examples.

      What are you waiting for? Let’s dig in.

      What is a Landing Page?

      A landing page is a web page where you send visitors with the goal to convert them into paying customers, subscribers, etc. It’s typically a standalone page, meaning it’s sole purpose is to receive traffic from your marketing campaigns and generate targeted leads with a focused Call to Action (CTA).

      Related: 7 Tips for Writing Winning Calls to Action for Your Website

      The best landing pages are simple and showcase benefits for the user, while helping a visitor to opt-in (like, to an email list) and convert them into buying customers or loyal subscribers. For example:

      Uber’s landing page is focused on getting users to sign up with a simple opt-in form.

      Slack’s A+ landing page focuses on building an email list with a bold CTA button.

      Think about it this way: you’ve invited a large crowd to an event. When people come, how are they received? What type of environment do they arrive at? Disorganized chaos will obviously turn crowds off at the door, while a clean and inviting space will encourage them to stay and engage.

      It’s the same with landing pages. How you receive traffic on your site determines who comes to stay — and what they do — on your site.

      In terms of grabbing the attention of audiences and generating leads, landing pages are one of the most important elements you need to get right. Already feeling overwhelmed? Don’t worry! We’ll demystify the elements of strong landing pages and how you can DIY-it on your site.

      Determine Your Objectives

      Sure, you want to drive traffic to your page. That’s important. Your site needs to attract eyes. But you also want visitors to do something on your site. Your landing page is the targeted platform that should make it simple — and compelling — for users to act, whether that be to buy, read, subscribe, or like.

      The truth is, landing pages sometimes get a bad rep. Website owners complain about low conversion rates from their pages, which is true: many landing pages don’t increase conversions. But, that doesn’t have to be your site’s reality. Your landing page can be a positive lead generator if done right.

      To begin, you really want to know what you’re after. Landing pages will differ based on industry, products or services being offered, your call to action and purpose, and of course, the people visiting the page — your audience. You want to figure out what your goals are so that you can design your landing page accordingly. And ideally, you’re creating a new landing page for each campaign. Most people don’t, even when it’s proven to increase conversion opportunities.

      Take a minute or two to map out what your objectives are — for your site as a whole (no fancy program needed; a spreadsheet should do the trick). Consider your audience and what their needs are (we’ll discuss this later). It might even be helpful to flesh out buyer personas.

      Also think about: what is the reason for the campaign in the first place? To build brand awareness? Increase sales? Establish a healthy email list? Whatever the reason, understanding your primary aim helps bring purpose and vision to your goal setting and the design of your landing page. With the overarching goal in mind, you can increase the chances of getting your visitors to do what you want them to. Score!

      Let’s start building our pages.

      6 Elements of Strong Landing Pages

      It’s true, there’s really no such thing as a one-size-fits-all landing page since your goals, offerings, and audiences are unique to your website. But luckily for you, landing pages with high conversion rates do have certain crucial characteristics in common.

      When in doubt, (or, if you’re just a landing page newbie) you can fall back on these tried-and-true elements before tailoring your page to meet your specific needs. A strong landing page must have:

      1. A Headline and Subhead

      Yeah, we know you checked out that catchy “32 Things You’ll Only Find Funny If You Loved ‘Stranger Things 2’” article when you were supposed to be responding to an email. And there’s a reason for that. The headline grabbed you, right?

      Now, you don’t need to be BuzzFeed or name drop the Kardashians to garner attention. The principle is this: if your headline is captivating, persuasive, benefit-driven, and appealing, you’ll have a much better chance of getting your audience to stick around and do something on your site.

      In fact, it’s crucial: more than 90 percent of visitors who read your headline also read your CTA copy, so it has to hook them if you want action. Your headline also largely affects your page’s shareability; thus, more eyes equals better chances of gaining conversions.

      The headline is easily the most important element of your landing page. It’s the first thing readers will see, and with attention spans at an all-time low, you’ve got less than ten seconds to captivate your audience before they try another site.

      Yep, that’s all the time you have. (No pressure.)

      If you want to increase conversions, your headline is a great place to start, offering about 80 percent of the opportunity for improvement to boost rates.

      When crafting your headline, make sure that it is:

      • Clear
      • Creative
      • Focused on benefits
      • To the point
      • Matching with ad text

      Take a look at these smart landing pages:

      Hello Fresh’s headline quickly and clearly identifies a benefit to their meal-kit delivery service — an easy dinner option for people who don’t have the time to grocery shop. Bam, we’re hooked.

      WordPress’ landing page headline is captivating and creative. Twenty-nine percent of the internet? Wow. We need to know more.

      A winning headline might not happen on the first try, and you may need to test more than one headline option to see which ones are the most effective for your landing page (more on testing later).

      Don’t know where to start?

      Experiment with a tried-and-true formula to get the juices flowing. Or, craft your headline last, after you’ve written copy and developed other page elements — then you might have a better idea of what you want to convey.

      Also consider the formatting of your headline. Remember: Periods. Stop. Eyes.

      You want your headline to flow easily for readability, so when it comes to punctuation, use sparingly. If you need to give “eye rests” or visual spaces for design purposes, try an em dash ( — ) or ellipses. Use the title case and make sure that your headline is big, bold, centered, and distinguishable in an above-the-fold location.

      Also, make sure that it matches your ad copy. Whether a visitor lands on your page via a social media push or an email campaign, they should know what to expect.

      Now to the subhead. Think of it like the Robin to your headline’s Batman.

      Like a great sidekick, a subhead adds further detail and context to your headline. It helps to funnel your readers down your content and gives them a reason to keep reading. A headline and subhead make a great — and essential — team, so don’t forget to pair up.

      Stitch Fix’s subhead provides additional context and continues to build interest in the product.

      A great subhead contains information that is relevant and to the point (this is not a place for distracting fluff). Communicate the value you offer and make every word count.

      2. Body Copy

      If readers have made it past both headline and subhead to your copy, give yourself a high five. You’ve created quality titles, and you’re off to great start.

      Now, your body copy: another key element that helps to explain your offerings and build interest.

      Like the other aspects of your page, it has to be tailored to your overall goal. If your CTA is higher risk — say, committing to a trial subscription — you’ll probably need a longer body copy to adequately explain the benefits of your offer. In some cases, longer-length forms have proven to increase conversions.

      On the other hand, if you’re just requesting an email, you can probably get away with shorter, bite-size-piece copy. It just depends on what you’re selling and what promotes a positive user experience.

      Graze’s bite-size copy is easy to skim and chock-full of benefits.

      Regardless of length, copy should be easy to skim (remember our short attention spans?) so:

      • Break text up with bullets or shorten paragraphs into digestible chunks if necessary to increase readability.
      • Highlight value by working to answer the question: what’s in it for me? Sell an idea, not just a product or service.
      • Use urgency (like a sale countdown timer or number of items remaining) or a discount to encourage hasty action.

      Remember to choose your words carefully, and give the copy cognitive flow — our brains like logical, wrapped-with-a-bow endings. And don’t skip editing steps. Have a second pair of eyes scan for errors.

      3. Images and/or Video

      What do paintings like Starry Night or photographs like Migrant Mother teach us? For one thing, it proves that visuals are powerful.

      They are processed by the brain faster than text, so they’re easily going to be one of the first things noticed on your landing page. Quality visuals, whether they be illustrations, product images, graphics, screenshots, photographs, or videos are essential for any strong landing page.

      And more than the science behind it, visuals play an important role in establishing an emotional connection. Take videos for example. Incorporating them on landing pages can increase conversions by 86 percent.

      Videos are persuasive and can accomplish a hefty list of challenges, like promoting retention (helping visitors remain on your page longer), increasing trust, and giving your brand a voice. Plus, you can actually show your visitors how to use your product or service. And it’s no secret: users like videos over lengthy text. Give the people what they want and include video elements on your landing page.

      With Storyline, visitors aren’t dredging through loads of copy. Rather, they’re presented with a simple and appealing video which makes staying engaged easy.

      Consider how different audiences might respond to your visuals, keeping in mind that one type might appeal to them more than another. But whatever type you use, your page’s visuals should be relevant to your product/service and to other page elements. Stay authentic and on brand.

      4. Trust-Building Elements

      If you want your landing page to start producing higher conversions, it’s wise to build the confidence of your audience. These virtual trust-building exercises might include:

      • Reviews/testimonials from current clients/customers
      • Positive press blurbs
      • Real-time social proof (“So-and-So from City XYZ just bought Product A”)
      • Logos of popular clients/users
      • Security badge icons
      • Smart, creative design (more on design to come)

      You can display these on your site to create a positive user experience and one that promotes a safe digital environment. Plus, it boosts your brand image.

      Edible cookie dough company Edoughble proudly displays the media outlets that have showcased their product. I mean, if Jimmy Fallon likes it, shouldn’t we?

      And in the case of testimonials, you’re infusing your landing page with voices of actual customers, which helps foster authenticity and makes human connections for those new or on-the-fence visitors.

      Keurig makes customer’s lauding a key part of its landing page, using a human connection to encourage buying.

      5. Clickable Call-to-Action

      Yeah, text, headlines, and photos are all great, but if there’s no way for your audience to actually do or click something on your page, it’s all in vain. 

      To boost conversions and start generating quality leads, you need a clear CTA. Whether it’s a shiny button, an easy-to-use opt-in form, or a colorful link, your page should have a CTA that’s easy to recognize and use (and it should be mobile friendly, folks).

      The whole goal of your page is to get visitors to do something. Your CTA is the means.

      You can spot Trello’s CTA a mile away. It’s clear, distinguishable, and a complement to the rest of the page — in both design and benefit offering.

      CTAs should be clearly distinguished on your site; they should be big, bold, and compelling. Give it an above-the-fold spot and a contrasting color for quick spotting. Also, make sure to test out phrasing options for your CTA buttons — simply inserting “FREE” or relying on the usual “Sign Up” won’t translate to automatic conversions for every landing page.

      6. Contact/Social Buttons

      An oft-overlooked element: further ways to engage.

      While you don’t want to incorporate any items on your landing page that distract visitors from completing the desired action, you can promote increased engagement and shareability by providing contact buttons and icons that link to your social media platforms.

      Chronicle Book’s landing page earns an A+ for contact and social buttons. We can easily find social media icons to lead us to more content and easy contact info.

      Social icons can act as free marketing — visitors have an easy way to share what they find with their own network and give you more exposure. Score!

      Contact buttons increase trust with your audience as you visibly make yourself available to their concerns and questions. As usual, test these elements to determine effectiveness on your page.

      Now that you know what elements make up a successful landing page, let’s get down to the nitty-gritty: the how of designing a landing page that converts.

      The 3 Core Principles of Landing Page Design

      Now, whether you’re a developer whose coding your landing page into existence or you’ve decided to save time by picking from a range of pre-made templates, you need to be aware of the principles that guide strong design.

      Along with a captivating headline, killer copy, and a powerful CTA, your design plays a big part in conveying your message. And even though we know not to judge a book by its cover, most audiences will judge the appearance of your landing page. So don’t underestimate the power of design, and award it the attention it deserves.

      Have we hammered that idea into your brain enough? Moving on. Here are some design principles to guide your building.

      1. Navigation/UX

      We know you don’t want to hear this, but we have to say it: it’s not about you. Welp. We know. But when it comes to the design of your landing page, every visitor is an opportunity, so it has to be all about them. The elements you include, the ones you don’t, and the way they’re arranged have to be oriented to the audiences who interact with your site.

      It needs to be a simple, straightforward, and pleasant experience for them, or they’ll ditch you for another site — probably your competition. Design every element with them in mind.

      Return again to your campaign’s main goal:

      • Is it to build an email list?
      • To increase sales?
      • To bolster your social following?
      • Improve brand awareness?

      Whatever it is, do it well. It should be crystal clear for visitors to come to your page what their next steps should be. Your interface should be simple, clear, and guide users to action — on the desktop and on mobile.

      Make all elements user-friendly; this includes type, links, visuals, menus, and buttons. And remember: your landing page elements shouldn’t be placed willy-nilly on a page. Rather, they should be consciously constructed for optimum impact.

      MailChimp’s interface is clean and user-friendly. Their goal is clear.

      To help users stay on your page and engage, limit navigational elements. You’ve brought audiences there — don’t encourage them to leave by placing distracting links on your page. You want them to act on your landing page CTA, so keep exit options limited and implement a bare (or shall we say, simple) interface.

      Lastly, consider that web audiences typically read in an F-shaped pattern. Let this telling shape principle guide where you place page elements.

      2. Visuals

      Even with handy landing page templates, there is plenty of room for creativity within your design. Plus, good design is essential for guiding your visitors through your page to complete the desired action.

      A well-constructed page design contributes to high lead generation, while poor design can confuse, frustrate, and even repel visitors.

      Use design to create a path for the eye. Users shouldn’t have to think about where to go next. The design should automatically lead them through. Use images and design elements to guide viewers’ eyes to your CTA.

      We process Warby Parker’s visual first and then are drawn to the CTAs. Even with very little text, the benefit is clear. This example showcases how design is impactful in generating leads.

      3. Colors

      Think of your favorite brands or products. You probably remain a loyal customer because you associate their products or services with pleasant emotions or feelings. And often, those good vibes are built through the use of color. Don’t just take our word for it. It’s true: there’s a whole psychology behind it.

      Think of music streaming service Spotify:

      Its colors evoke calmness, peace, a “chill” vibe, — moods typically associated with listening to good music. Blue is also a popular color for marketing and sales efforts, as it often stimulates feelings of security and stability. Colors can be extremely powerful in getting people to respond and act.

      Whether they realize it or not, your visitors are being affected by the colors implemented in your design. It can encourage them — subconsciously, even — to feel certain emotions and be more likely to respond in certain ways.

      It’s true: in a study, researchers found that 90 — yes, 90! — percent of consumer decisions about products are based on color. Yep, it’s that important. Effective use of color will improve your landing page conversions.

      In addition to building an environment, it’s important to be smart about your user of color as it can largely affect user experience and brand awareness, thus, affecting your overall conversion goals.

      Related: How to Brand Your WordPress Theme — Tips for Making a Template Work for You

      Some additional hints:

      • Your CTA button is the most important element you can differentiate by color. It needs to be an obvious, clear, and an attractive standout on your page, while still conveying the right emotion for your message. Test out various color schemes and find what works best.
      • Be aware of background colors, as you don’t want to obscure text, visuals, or other page elements. And don’t be afraid of white! White backgrounds can give your site a minimalist, clean feel and highlight key pieces of information.
      • It’s not really wise to have outbound links on your landing page — hey, you want users to stick around on your site. But if you do, be aware of link color.
      • While there are guidelines for color psychology and tried-and-true landing page design templates, there really are no hard-and-fast rules for color (or many other elements of landing page creation), so test out what works for you. If popular sites have blue backgrounds, but that doesn’t work for you, toss it. Industries and audiences vary. Appeal to your users. And always: test, test, test.

      6 Landing Page Best Practices to Keep in Mind

      Wow. Your page is coming along great! But wait — there’s more to learn. Here’s a handful of landing page best practices to pay attention to as you continue to fine-tune your pages.

      1. Do Your Homework

      Bleh, homework? Don’t worry, this type is fun! After you’ve established your goals for a new marketing campaign, do some research. You’ll automatically create stronger landing pages if you take the time to collect information about your target customers and audiences. You’ll also want to study up on the popularity of your topic and trends over time.

      Armed with this vital knowledge, you can better tailor your landing pages to provide the highest value and most positive UX for your visitors. Some tools for conducting market research:

      Customer Profiles

      • Remember buyer personas? This HubSpot tool helps you build them to visualize your customer profiles.
      • Want to get an idea of who might like your product? Create a mockup profile of customers who like certain products or brands with YouGov Profiles.

      Feedback

      • Don’t wait for customers to come to you. Go ask them for direct feedback with survey-building tools like SurveyMonkey, AYTM, or FieldBloom. Bonus: it’s budget-friendly and helps you improve your brand and UX.

      Trends

      • What’s popular in internet conversation? #WorldWaterDay? #MeToo? Explore trends in your topic over time in comprehensive detail with Google Trends.
      • Want to put a finger on the pulse of social conversation? Analyze real-time social media trends by monitoring on tools like HootSuite.
      • Research the popularity of certain keywords in your industry and discover intent by utilizing Google’s Keyword Planner. Keyword intent research helps you understand what users really want, not just the phrases and words they use to search. That’s key insight.

      Data

      • Clicks heard across the world: let Google help you understand how the world is using the internet with its Consumer Barometer.
      • Need some scholarly stats? Access raw datasets in a host of fields from the Pew Research Center.
      • Think with Google is a beneficial tool from the web giant designed to share useful stats, tools, trends, and insight. An example? Explore shopping trends and product popularity with Google’s Shopping Insights.

      Remember, when it comes to building strong landing pages and increasing conversions, ignorance is not bliss. Get smart.

      2. Make it Fast

      Sad, but true: the one thing your site visitors don’t have is patience. Your visitors aren’t going to stick around forever — in fact, a measly one second delay in page speed can decrease conversions by seven percent. Slow speeds kill conversions.

      On the other hand, mobile pages that loaded only one second faster were able to increase their conversion rate by 27 percent. That can dramatically affect your business in a big way.

      Yes, load times on your site are that important — not only for optimal page design but for increasing conversions. There could be many culprits to blame for your sluggish site, like bulky code or unoptimized images. Unless you want a mass exodus, you’ve got to speed things up. Luckily, you’ve got us. As your web hosting gurus, we know a thing or two about how to pick up the pace. Try these tips:

      1. Choose a good hosting provider (Check! We’ve got your back).
      2. Optimize images.
      3. Minify your resources like JavaScript and CSS.
      4. Leverage caching.
      5. Get rid of outdated or unused plugins.

      After implementing these simple fixes, you’ll want to continue to test your site often. Google’s PageSpeed Insights can help with that. Get key info about the real-time performance of your site and suggestions for optimization.

      Slow and steady won’t win this race.

      3. Be Responsive

      No, we don’t mean like answering your backup of emails or text messages (though, that’s important, too). Being responsive means having a landing page that functions seamlessly across devices.

      More and more, having a responsive landing page is a deal breaker for web users — and for the success of your landing page. Currently, there are more mobile internet users than desktop users — a staggering 3.5 billion globally as of August 2017. If you’re not catering to those mobile visitors (only 50 percent of companies are), you’re cutting off your opportunities for increased conversions.

      With audiences turning more to their smartphones and tablets to shop and make purchases, you need to think mobile in a big way. Make sure your landing page is accessible (with all parts functional) on any device — whether your target users are riding the subway or checking out e-deals on their lunch break — and you can triple your chances of increasing conversions by 5 percent (or more).

      If you’re using a handy landing page-building software, be sure to outfit your site with mobile-ready templates. Then test: you can use Google’s program or MobileTest.me.

      Always concern yourself with how to make your landing page more mobile-friendly. You’ll see the returns with increased conversions.

      4. Be Magnetizing

      We’ve talked a lot about how every element on your landing page should be geared toward getting your visitors to take action.

      Now, it’s time for a new vocab word to add to your marketing repertoire: lead magnet. What is it? Well, it’s useful (free) content that you give away to offer value to site visitors. Maybe you haven’t heard the term before, but you’ve definitely seen examples of it:

      Floret Flower company makes sure that subscribing to their email list is worth your while by offering a lengthy ebook in exchange.

      A good lead magnet focuses on value offered and not overly on your company or your product or service. It encourages users’ clicks and engagement and is constantly improving your brand. Even if sales or other rewards aren’t immediate, it’s a profitable practice.

      Because you’re offering valuable content, users are more willing to give up their info — say, their email for an ebook download — thus, helping you towards your goal of increased conversions.

      Does your landing page make a compelling offer? If not, edit it to showcase more value.

      5. Address Your Audience’s Pain Points

      Ideally, you want to be in the business of relieving pain. No, you don’t need to brandish bandaids or stock up on gauze. The kind of pain we’re talking about has more to do with the psychology of consumers and their clicks, aka pain theory.

      Whether it’s frustration over paying too much for cable or the lack of gluten-free meal options, people are looking for solutions to their most aggravating problems. To influence your audience and increase conversions, you must understand and solve their pain points.

      The Promptly Journals company is capitalizing on a simple pain of parents: forgetting the details of life. The desire to treasure the small moments of everyday urges visitors to shop.

      In your field, what nagging concerns might your visitors have? Cater to them by providing solutions (or products) to relieve those pains.

      • What keeps your audience up at night?
      • What do they worry about?
      • What would make their lives easier?

      Be there when they need you. Expend the effort to figure out what solutions you can offer them and do it well.

      You’ve showcased what they will gain when they answer your call, but what will they lose? Make it impossible to give up. After all, when you’re the answer to their problems, they’re likely to come back (and bring their friends).

      But a warning: make sure you understand their pain points and deliver what you’re promising. Few things irritate web users more than not receiving what they were promised. Don’t disappoint them.

      You can highlight how your product or service alleviates pain by showcasing real-person testimonials, embedding solutions in your benefit-and-optimism-ridden copy, or through impactful visuals.

      Pesky diaper leaks and sensitive skin: they’re pain points for many parents. The Honest company highlights how their diapers offer the solution with a real-life customer testimonial.

      6. Be (Very) Specific

      Ever stood in an aisle of the grocery store, staring at shelves of endless items, unable to choose a product — say, a type of cereal or brand of cookie — because of the multitude of choices? We all have.

      This dilemma is encapsulated in a principle called the Paradox of Choice — the simple truth that the more choices consumers have, the harder it is to make a decision. They might even leave the store empty-handed, too overwhelmed to choose.

      The same applies to landing pages: Even though 48 percent of landing pages out the web contain multiple offers, the strongest landing pages focus on one specific ask.

      Their purpose is to get visitors to act, not wade through a slew of daunting options. This overwhelms users and decreases your chances of converting those visitors into leads. If you try and pull double duty, conversions will be low.

      So remember, when it comes to CTAs, less is more. Focus on promoting one thing, and be specific, based on your goals. Generic asks aren’t effective; specific CTAs help build trust with your audience as they come to see your focused efforts as authoritative in your field.

      Hulu’s landing page goal is very specific: start a free trial of its streaming service. Every part of the page leads toward it, and there are few other elements to distract visitors from action.

      Make sure your design caters to promoting your specific CTA and remove elements that distract or divert attention. The design’s purpose is simple: get your site visitors to act on your CTA, whether it be clicking a button, downloading an ebook, filling out an opt-in form, or following your social media platforms (or whatever you’re asking for). Every element on your page should be aligned and directed toward your one purpose.

      Resources and Tools to Design Your Landing Page

      Now, it’s not required that you hire an expensive squad of designers, copywriters, or developers to ace your landing page game. We know not everyone has the budget to hire a team to build custom pages for your site. If your purse strings are tight, we’ve got some top-tier tools to help you DIY-it (no hot glue guns or glitter necessary).

      Just like you wouldn’t use scissors to cut your grass, you need the right tools to create successful landing pages. Using the wrong tools can even hinder your lead-generating results, while the right tools can help you effectively automate tasks (yay, less stress for you), streamline your website processes, and ultimately, increase conversions. That’s the goal, right?

      Here is our virtual toolbox of essential instruments, divided by category. Try them out and see what works best for you — and your budget.

      Design Tools

      Guess what? It’s possible to set up a landing page on your current site using WordPress, and it doesn’t take an advanced programming degree to do. Or, if that’s not what you’re looking for, try our Remixer site builder to craft a customized one-off page quickly and easily.

      Copy Resources

      So you haven’t really worried about grammar since college. That’s OK! Writing tools on the web exists for a reason, so use them!

      To write better copy, try Hemingway, a desktop app that acts as your digital editor, helping to make your writing more clear and powerful (like warning you to avoid that pesky passive voice). Grammarly is another popular proofreading option that you can install to your Chrome browser to help you write error-free wherever you write (even when crafting a Facebook post!).

      If you’re missing that classroom feel, you can hone your writing skills by signing yourself for a Udemy copywriting course — flexible e-classes you can take on your (busy) schedule.

      Visuals Resources

      In terms of professionalism, your own photography is best. But there are still many options for free, copyright-safe photos on the web that can provide you with vibrant and impactful graphics — a must have for a killer landing page. A host of good options:

      Related: Edit Your Website Photos Like a Pro with These 8 Tips

      CTAs/Opt-in Tools and Plugins

      Whether you’re implementing a pop-up opt-in or a simple email registration box, there’s a tool available to make your life — and the design of your landing page — easier.

      There are tons of tools out there, so do your research and compare options before committing to an opt-in service. Look for a tool that lets you customize design and track performance. And take into consideration that many email marketing service providers often have their own suite of tools. For you WordPress users, we recommend OptinMonster, one of the best (and most popular) options on the market. It offers a variety of campaign choices so you can customize it to your needs.

      Before you go po-up crazy, though, check out our guide about the dark-side of UX design. You don’t want to scare off visitors with shady tactics.

      How to Measure Your Landing Page Success (4 Metrics)

      Lucky for you, focusing your landing page on one specific goal often makes analytics simpler. So how do you know if your landing page is effective at increasing conversions? What numbers are the most important to focus on when measuring success?

      For advanced users, there are many metrics you can track to give you insight on your page’s performance. But for those of you just getting starting, begin by keeping tabs on the following metrics:

      1. Conversion Rate

      This is a no-brainer, of course. The conversion rate measures how often visitors are answering your ask, whether that be buying something, subscribing to your email list, or following a social platform.

      As far as metrics go, this is probably the most important. After all, that’s what your landing page is trying to accomplish: more conversions.

      Note: the average conversion rate is just over 2 percent. The highest converting landing pages often reach 11 percent or higher.

      2. Traffic Source

      What platforms are driving visitors to your site? A Facebook ad? A Google search? Referral from another blog? Determining traffic sources can help you understand where to spend your marketing budget and how to optimize other aspects of your landing page, like increasing responsiveness.

      3. Form Abandonment Rate

      As you utilize opt-in forms on your landing page, you’re asking visitors to provide you with information, like their name and email address. The form abandonment rate is the frequency at which users are leaving these opt-ins without completing the necessary fields.

      If this rate is high, you might be asking for too much information and driving away leads. Test opt-in forms to determine the ideal number of fields that visitors are willing to fill out in exchange for your lead magnet or offering.

      4. Time on Page

      How long are users spending on your site? Understanding this number can help you understand how engaging your content is, and how likely visitors are to convert.

      Another word on testing: you might be shocked to learn that only half of companies and agencies that use landing pages test them. HALF. To find ways to continually improve your pages and increase conversions, you need to test.

      Most analytics programs will allow you to test different versions of your site, and this is key. Metrics clue you into what you need to change, and by conducting A/B split tests, you can determine how to implement those changes.

      Related: A/B Testing — How to Improve Your Website Through Trial and Error

      A/B testing is the most widely-used improvement method and involves creating two versions of a page to evaluate how one key element changed between the two makes a difference, and which is more effective. It can also help you get to know your audiences better and what their web preferences and behaviors are.

      Did You Know? The 2008 Obama campaign made an additional $60 million by conducting A/B testing on elements of their website!

      Keep refining your landing pages so they are as effective as possible. This will take time and some work, but it’s well worth the effort. Just like it takes several drafts to polish prose, it’s likely that you won’t get a blockbuster lead-generating landing page on the first try. These things require tweaking (and more tweaking) to get right. And they’ll need to be tested for each campaign you run.

      But don’t worry — there are tools for that.

      For A/B testing and analytics, utilize Google Analytics, Optimizely, VWO, or Kissmetrics. Crazy Egg can help you see what specific parts of your website users are interacting with, which can help you improve design.

      DIYing your stats? Here’s a math guide to help crunch those numbers.

      Stick the Landing

      We’ll bite on it: you’re never going to visit another website without first stopping to evaluate the landing page, are you? These often-underutilized pages have the power to turn page visitors into customers and loyal subscribers.

      Yep, they’re powerful — if done right.

      Combine a captivating headline, killer copy, engaging visuals, and powerful CTA with smart design, and you’ve got a solid recipe for higher conversions. No luck required.



      Source link