One place for hosting & domains

      Programming

      Learn the AWK Programming Language


      Updated by Linode Contributed by Mihalis Tsoukalos

      What is AWK?

      AWK is a Turing-complete pattern matching programming language. The name AWK is derived from the family names of its three authors: Alfred Aho, Peter Weinberger and Brian Kernighan. AWK is often associated with sed, which is a UNIX command line tool. However, sed is more appropriate for one line UNIX shell commands and is typically used only for text processing.

      AWK is great for data reporting, analysis, and extraction and supports arrays, associative arrays, functions, variables, loops, and regular expressions. Current Linux systems use improved versions of the original AWK utility. The main enhancement to these AWK variants is support for a larger set of built-in functions and variables. The most widely used variants of AWK are: Gawk, Mawk, and Nawk.

      Note

      This guide uses the Gawk version of AWK.

      There are many practical uses of AWK. For example, you can use AWK and the history command to find your top 10 most frequently issued commands:

      history | awk '{CMD[$2]++;count++;} END {for (a in CMD)print CMD[a] " "CMD[a]/count*100 " % " a;} ' | grep -v "./" | column -c3 -s " " -t | sort -rn | head -n10
      

      This guide assumes familiarity with programming language concepts and is meant to provide an overview of some basic elements of the AWK programming language. In this guide you will learn:

      AWK Basics

      In this section you will learn basics of the AWK programming language, including:

      • How to execute AWK from the command line with one-off commands and by storing AWK code in files.
      • Creating and using variables, arrays, and functions.
      • Special patterns, like BEGIN and END.

      Note

      A pattern in AWK controls the execution of rules and a rule is executed when its pattern is a match for the current input record.

      Run an AWK Program

      A program in AWK can be written via the command line or by executing a file containing the program. If you want to reuse your code, it is better to store it in a file. AWK reads input from standard input or from files specified as command line arguments. Input is divided into individual records and fields. By default, new lines are parsed as a record and whitespace is parsed as a field. After a record is read, it is split into fields. AWK does not alter the original input.

      The next two sections will walk you through creating a Hello World! program that you will run, both as a one-off program on the command line, and as reusable code saved in a file.

      Hello World! – Command Line

      When an AWK program contains the BEGIN pattern without another special pattern, AWK will not expect any further command line input and exit. Typically, when an AWK program is executed on the command line, without the BEGIN special pattern, AWK will continue to expect input until you exit by typing CTRL-D. The example Hello World! program below will print and immediately exit.

      1. Execute the command as follows:

        awk 'BEGIN { print "Hello World!" }'
        

        The output will be as follows:

          
        Hello World!
        
        

      Hello World! – Input File

      In this section, you will create an AWK program in an input file and then run it from the command line.

      1. Create a file called helloworld.awk with the following content:

        helloworld.awk
        1
        2
        
        BEGIN { print "Hello World!" }
            
      2. On the command line, run the helloworld.awk program. The -f option tells AWK to expect a source file as the program to run.

        awk -f helloworld.awk
        
      3. The output will be as follows:

          
        Hello World!
            
        
      4. You can also run AWK programs as executable scripts. Open helloworld.awk and add a bash script line to the top of the file and save it without the .awk extension.

        helloworld
        1
        2
        3
        4
        
        #!/usr/bin/awk -f
        
        BEGIN { print "Hello World!" }
            

        The #!/usr/bin/awk -f line defines the start of script execution.

      5. Add execute permissions to helloworld:

        chmod +x helloworld
        
      6. Execute the helloworld program:

        ./helloworld
        
      7. The output will resemble the following:

          
        Hello World!
            
        

      Variables in AWK

      AWK supports built-in and user defined variables. Built-in variables are native to AWK, whereas user defined variables are ones you define.

      Built-in Variables

      AWK has many built-in variables that are automatically initialized. Some of the most important ones are the following:

      Variable Definition
      NF Holds the number of fields in the current input record. Each record can have a different number of fields.
      FS Defines the input field separator. The default value is a whitespace, but it also matches any sequence of spaces and tabs. Additionally, any number of leading or trailing whitespaces and tabs are ignored. If the value of FS is set to the null string, then each character in the current line becomes a separate field.
      FILENAME Stores the filename of the current input file. You cannot use FILENAME inside a BEGIN block, because there are no input files being processed.
      NR Keeps track of the total number of records that have been read so far.
      FNR Stores the total number of records that have been read from the current input file.
      IGNORECASE Tells AWK whether or not to ignore case in all of its comparisons or regular expressions. If IGNORECASE stores a non-zero or null value, then AWK will ignore case.
      ARGC Holds the number of command line arguments.
      ARGV Stores the actual command line arguments of an AWK program.

      User Defined Variables

      User defined variables can store numeric or string values. AWK dynamically assigns variables a type based on the variable’s initial value. User defined variables, by default, are initialized to the empty string. If you convert a variable from a string to a number, the default value is zero. You can convert a string to a number and vice versa as long as the string can be converted to a valid number. It is important to keep in mind that AWK is not a type safe programming language, since this can sometimes generate bugs.

      • You can set a variable via the command line using the -v option. This command will initialize the variable count and print its value:

        awk -v count=8 'BEGIN { print count }'
        
      • To initialize variables within an input file, you can use the form myvariable = "myvar" for strings and myvariable = 10 for numeric values. Create a file named count.awk and add the following content:

        count.awk
        1
        2
        3
        4
        5
        
        BEGIN {
            count = 10
            print count
        }
            

        To run this file, switch back to the command line and execute the following command:

        awk -f count.awk
        

        Your output should display:

          
              10
            
        

      Special Patterns

      AWK uses patterns to control how a rule should be executed against an input record. The two main categories of patterns in AWK are regular expressions and expressions. Regular expressions use a special format to target specific sets of strings, while expressions encompass various ways to target patterns in AWK, like comparison expressions that may utilize regular expressions. Special patterns in AWK include reserved keywords that perform special actions within your AWK programs. The sections below discuss the special patterns BEGIN, END, BEGINFILE, and ENDFILE.

      BEGIN and END

      BEGIN and END are executed only once: before receiving any input and after processing all input, respectively. In this way, they can be used to perform startup and cleanup actions in your AWK programs.

      Although it is not required to use BEGIN and END at the beginning and end of your AWK programs, it is considered good practice to do so. Additionally, you can include multiple BEGIN and END blocks in one program.

      If an AWK program uses only BEGIN rules without any other code, the program terminates without reading any of the specified input. However, if an AWK program contains only END rules without any additional code, all the specified input is read. This is necessary in case the END rule references the FNR and NR variables.

      BEGINFILE and ENDFILE

      Note

      BEGINFILE and ENDFILE only work with gawk.

      Two other patterns with special functionality are BEGINFILE and ENDFILE. BEGINFILE is executed before AWK reads the first record from a file, whereas ENDFILE is executed after AWK is done with the last record of a file.

      ENDFILE is convenient for recovering from I/O errors during processing. The AWK program can pass control to ENDFILE, and instead of stopping abnormally it sets the ERRNO variable to describe the error that occurred. AWK clears the ERRNO variable before it starts processing the next file. Similarly, the nextfile statement – when used inside BEGINFILE – allows gawk to move to the next data file instead of exiting with a fatal error and without executing the ENDFILE block.

      1. As an example, create a file named beginfile.awk:

        beginfile.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        
        BEGIN {
            numberOfFiles = 0
        }
        
        BEGINFILE {
            print "New file", FILENAME
        
            # Check if there is an error while trying to read the file
            if (ERRNO) {
                print "Cannot read", FILENAME, "– processing next file!"
                nextfile
            }
        }
        
        ENDFILE {
            numberOfFiles++
        }
        
        END {
            print "Total number of files processed: ", numberOfFiles
        }
        • This program showcases the usage of BEGIN, END, BEGINFILE, and ENDFILE by printing the total number of files read as well as the filename of each file.

        • If there is a problem while reading a file, the code will report it.

        • Printing the filename is done with the help of the FILENAME variable.

      2. Execute the file with the following command:

        gawk -f hw.awk beginfile.awk givenLine.awk doesNotExist
        
      3. The output will be similar to the following example. The program does not stop abnormally when it does not find an input file and provides a useful error message.

          
        New file hw.awk
        Cannot read hw.awk – processing next file!
        New file beginfile.awk
        New file givenLine.awk
        Cannot read givenLine.awk – processing next file!
        New file doesNotExist
        Cannot read doesNotExist – processing next file!
        Total number of files processed:  1
            
        

      Looping in AWK

      AWK supports for, do-while, and while loops that behave similarly to control flow statements in other programming languages. Loops execute code contained within a code block as many times as specified in the control flow statement. To illustrate loops in AWK, a working example is provided below.

      1. Create and save a file named loops.awk:

        loops.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        
        BEGIN {
            for (i = 0; i < ARGC; i++)
                printf "ARGV[%d] = %sn", i, ARGV[i]
        
            k = 0
            while ( k < ARGC ) {
                printf "ARGV[%d] = %sn", k, ARGV[k]
                k++
            }
        
            m = 0
            do {
                printf "ARGV[%d] = %sn", m, ARGV[m]
                m++
            } while ( m < ARGC )
        }
        
        END {
            for (i = 0; i < 10; i++)
                printf "%d ", i
            printf "n"
        }
        • The program uses the value of the ARGC built-in variable to control how many times to loop through each separate block of code. The result will vary depending on how many command line arguments you pass to AWK when executing the program.
        • The for loop after the END special pattern will print numbers from 0 – 9.
      2. Execute the loops.awk input program with the following command:

        echo "" | awk -f loops.awk
        
      3. The output will be similar to the following:

          
        ARGV[0] = awk
        ARGV[0] = awk
        ARGV[0] = awk
        0 1 2 3 4 5 6 7 8 9
        
        

      Arrays

      AWK does not require array indices to be consecutive integers. Instead, strings and numbers may be used. This is because AWK uses string keys internally to represent an array’s indices, and so arrays in AWK are more like associative arrays that store a collection of pairs. Unlike other programming languages, you do not need to declare an array and its size before using it, and new pairs can be added at any time. The file below serves to illustrate the behavior of arrays in AWK.

      1. Create the file arrays.awk:

        arrays.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        
        BEGIN {
            a[0] = 1;
            a[1] = 2;
            a[2] = 3;
            a[3] = 4;
        
            for (i in a)
                print "Index:", i, "with value:", a[i];
                print "Adding two elements and deleting a[0]";
        
                a["One"] = "One_value";
                a["Two"] = "Two_value";
                delete a[0];
        
            for (i in a)
                print "Index:", i, "with value:", a[i];
        
            if (a["1"] == a[1])
                printf "a[1] = a["1"] = %sn", a["1"];
        }
        • The program creates the a[] array and initializes it with four separate numeric values.
        • The for block will loop through the array and print the current index and value.
        • It then adds two new elements to array a[] that use string indices instead of numbers.
        • It demonstrates how to delete an element from an array by deleting the a[0] element.
        • Finally, the if statement evaluates if a["1"] and a[1] are equivalent. Since AWK stores all array elements as string keys, both indices point to the same array element and the code in the if statement executes.
      2. Run the program with the following command:

        awk -f arrays.awk
        
      3. The output will look similar to the following:

          
        Index: 0 with value: 1
        Index: 1 with value: 2
        Index: 2 with value: 3
        Index: 3 with value: 4
        Adding two elements and deleting a[0]
        Index: Two with value: Two_value
        Index: One with value: One_value
        Index: 1 with value: 2
        Index: 2 with value: 3
        Index: 3 with value: 4
        a[1] = a["1"] = 2
          
        

        Note

        The order of the array indices may be out of order. This is because arrays in AWK are associative and not assigned in blocks of contiguous memory.

      Functions

      Like most programming languages, AWK supports user-defined functions and ships with several useful built-in functions. This section will provide examples demonstrating how to use both types of functions.

      Predefined Functions

      AWK’s built-in functions provide mechanisms for string manipulation, numeric operations, and I/O functions to work with files and shell commands. The example below utilizes the built-in numeric functions rand() and int() to show how to call built-in functions.

      1. Create and save a file named rand.awk:

        rand.awk
        1
        2
        3
        4
        5
        6
        7
        
        BEGIN {
            while (i < 20) {
                n = int(rand()*10);
                print "value of n:", n;
                i++;
            }
        }
        • The rand.awk program uses the rand() function to generate a random number and stores it in the n variable. By default, rand() returns a random number between 0 and 1. To generate numbers larger than 1, the program multiplies the returned random number by 10.
        • AWK’s int() function rounds the result of the rand() function to the nearest integer.
      2. Execute the rand.awk program with the following command:

        awk -f rand.awk
        
      3. The output will resemble the following:

          
        value of n: 2
        value of n: 2
        value of n: 8
        value of n: 1
        value of n: 5
        value of n: 1
        value of n: 8
        value of n: 1
        ...
        
        

      User Defined Functions

      The AWK programming language allows you to define your own functions and call them throughout an AWK program file. A function definition must include a name and can include a parameter list. Function names can only contain a sequence of letters, digits, and underscores. The function name cannot begin with a digit. In the example below, you will declare a function definition and utilize it within the AWK program.

      1. Create and save the myFunction.awk file:

        myFunction.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        function isnum(x) { return(x==x+0) }
        
        function sumToN(n) {
            sum = 0
            if (n < 0) { n = -n }
            if ( isnum(n) ) {
                for (j = 1; j <= n; j++)
                    sum = sum + j
            } else { return -1 }
            return sum
        }
        {
            for (i=1; i<=NF; i++)
                print $i, "t:", sumToN($i)
        }
        • The user defined function sumToN() takes a single parameter n and uses a for loop to increment its value and stores it in the sum variable.
        • The program will take command line input, and pass it as a parameter to the sumToN() function and print the calculated sum.
      2. Execute myFunction.awk with the following command:

        echo "10 12" | awk -f myFunction.awk
        
      3. Your output will resemble the example below. If you use a different set of numbers, your output will differ from the example.

          
        10 : 55
        12 : 78
        
        

      Practical Examples

      This section of the guide provides a variety of practical examples to further demonstrate the AWK programming language. You can try out each example on your own Linux machine or expand on the examples for your own specific needs.

      Printing

      Printing a Given Line from a File

      1. To use AWK to print a given line from a text file, create and save the givenLine.awk file:

        givenLine.awk
        1
        2
        3
        4
        
        {
            if (NR == line)
                print $0;
        }
        • This program will print out the record that corresponds to the value passed to the line variable. The program will require input either from the command line or from a file.
        • You should pass the value of the line variable to the AWK program as a command line argument using the -v option.
      2. By executing the givenLine.awk program as follows, it will print out the first line found in the myFunction.awk program written in the previous section. (You could similarly pass it any text file.)

        awk -v line=1 -f givenLine.awk myFunction.awk
        

        The output will resemble the following:

          
        function isnum(x) { return(x==x+0) }
        
        
      3. Execute givenLine.awk again, passing line 4:

        awk -v line=4 -f givenLine.awk myFunction.awk
        
      4. This time the output is as follows:

          
            sum = 0
        
        

      Printing Two Given Fields from a File

      In this example, the AWK program will print the values of the first and third fields of any text file.

      1. Create and save the file field1and3.awk:

        field1and3.awk
      2. Create and save the file words.txt:

        words.txt
      3. Execute field1and3.awk passing words.txt as input:

        awk -f field1and3.awk words.txt
        
      4. The output will print only the first and third words (fields) contained in the file:

          
        one three
        
        

        Note

        You can also execute the contents of field1and3.awk on the command line and pass words.txt as input:

        awk '{print $1, $3}' words.txt
        

      Counting

      Counting Lines

      The following example AWK program will count the number of lines that are found in the given text file(s).

      FNR stores the total number of records that have been read from the current input file.

      1. Create and save the countLines.awk file:

        countLines.awk
        1
        2
        3
        4
        5
        6
        7
        8
        
        {
            if (FNR==1)
                print "Processing:", FILENAME;
        }
        
        END {
            print "Read", NR, "records in total";
        }
        • The use of FNR makes sure that the filename of each processed file will be printed only once.
        • END makes sure that the results will be printed just before AWK finishes executing countLines.awk.
      2. Create and save the data.txt file. This file will be passed to AWK as input for processing.

        data.txt
        1
        2
        3
        4
        5
        6
        7
        
        one
        two
        three
        4
        
        6
        seven not eight
      3. Execute countLines.awk with the following command, passing data.txt as input:

        awk -f countLines.awk data.txt
        
      4. The output will resemble the following:

          
        Processing: data.txt
        Read 7 records in total
        
        
      5. Execute countLines.awk with multiple files for processing. You can use words.txt from the previous exercise.

        awk -f countLines.awk data.txt words.txt
        
      6. You should see a similar output:

          
        Processing: data.txt
        Processing: words.txt
        Read 8 records in total
        
        

      Counting Lines with a Specific Pattern

      The following AWK code uses the variable n to count the number of lines that contain the string three:

      awk '/three/ { n++ }; END { print n+0 }'
      
      • The code above tells AWK to execute n++ each time there is a match to the /three/ regular expression.

      • When the processing is done, the code in END is executed. This code prints the current value of n converted to a number by adding the numeral zero.

      1. Create a file named dataFile.txt to pass to AWK as input for processing:

        dataFile.txt
        1
        2
        3
        4
        5
        6
        7
        
            one
            two
            three
            four
            three
            two
            one
      2. Execute the example code and pass dataFile.txt as input:

        awk '/three/ { n++ }; END { print n+0 }' dataFile.txt
        
      3. The output will look as follows:

          
        2
        
        

      Counting Characters

      In this example, the countChars.awk file calculates the number of characters found in an input file.

      1. Create and save the file countChars.awk:

        countChars.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        
        BEGIN {
            n = 0;
        }
        
        {
            if (FNR==1)
                print "Processing:", FILENAME;
        
            n = n + length($0) + 1;
        }
        
        END {
            print "Read", n, "characters in total";
        }
        • This program makes use of the built-in string function length(), which returns the number of characters in a string. In the case of the program, the string will be provided by the entirety of the current record, which is indicated by $0.
        • The + 1 appended to the length() function is used to account for the new line character that each line includes.
      2. Execute countChars.awk by running the following command and pass it the countLines.awk file from the previous exercise.

        awk -f countChars.awk countLines.awk
        
      3. The output will look similar to the following:

          
        Processing: countLines.awk
        Read 110 characters in total
        
        
      4. Execute countChars.awk with multiple files to process as follows:

        awk -f countChars.awk countLines.awk field1and3.awk
        
          
        Processing: countLines.awk
        Processing: field1and3.awk
        Read 132 characters in total
        
        

      Calculating Word Frequencies

      This example demonstrates some of the advanced capabilities of AWK. The file wordFreq.awk reads a text file and counts how many times each word appears in the text file using associative arrays.

      1. Create and save the file wordFreq.awk:

        wordFreq.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        {
            for (i= 1; i<=NF; i++ ) {
                $i = tolower($i)
                freq[$i]++
            }
        }
        
        END {
            for (word in freq)
                print word, ":", freq[word]
        }
        • wordFreq.awk uses a for loop to traverse through an input file and add each record to the freq[] array.
        • The tolower() built-in string function is used to ensure the program does not count the same word multiple times based on differences in case, e.g., seven and Seven are not counted as different words.
        • Before the program exits, the END block prints out each word and its frequency with the input file.
      2. Create and save the file wordFreq.txt to use as an input file.

        wordFreq.txt
        1
        2
        3
        4
        5
        
        one two
        three one four seven Seven
        One Two TWO
        
        one three five
      3. Execute the wordFreq.awk program and pass wordFreq.txt as input:

        awk -f wordFreq.awk wordFreq.txt | sort -k3rn
        

        Note

        The sort -k3rn command is used to sort the output of wordFreq.awk based on a numeric sort in reverse order.

      4. The output will resemble the following:

          
        one : 4
        two : 3
        seven : 2
        three : 2
        five : 1
        four : 1
        
        

      Updating Docker Images

      1. Use the following series of piped commands to update all Docker images found on your local machine to their latest version:

        docker images | grep -v REPOSITORY | awk '{print $1}' | xargs -L1 docker pull
        
        • In this example, AWK is just a piece of the entire command. AWK does the job of extracting the first field from the result of executing the docker images command.

      Finding

      Finding the Top-10 Commands of your Command History

      1. Use The following shell command to find your top 10 most used commands by piping the output of history to AWK as input:

        history | awk '{CMD[$2]++;count++;} END {for (a in CMD)print CMD[a] " "CMD[a]/count*100 " % " a;} ' | grep -v "./" | column -c3 -s " " -t | sort -rn | head -n10
        
        • First, the command executes the history command to be used as AWK’s input.

        • This is processed by a complex awk command that calculates the number of times each command appears in history by considering the second field of each record. This is the field that corresponds to the previously issued commands. These values are stored in the CMD[] associative array.

        • At the same time, the total number of commands that have been processed are stored in the count variable.

        • The frequency of each command is calculated with the CMD[a]/count*100 statement and printed on the screen along with the command name.

        • The formatting and the sorting of the output is handled by the grep, column, sort, and head command line utilities.

      2. Your output should resemble the following:

          
        2318  18.4775    %  git
        1224  9.75688    %  ll
        1176  9.37425    %  go
        646   5.14946    %  docker
        584   4.65524    %  cat
        564   4.49582    %  brew
        427   3.40375    %  lenses-cli
        421   3.35592    %  cd
        413   3.29215    %  vi
        378   3.01315    %  rm
        
        

      Finding the Number of Records that Appear More than Once

      This program’s logic utilizes the behavior of AWK associative arrays. The associative array’s keys are the entire lines of the passed input. This means that if a line appears more than once, it will be found in the associative array and will have a value that is different from the default, which is 0.

      1. Create and save the file nDuplicates.awk:

        nDuplicates.awk
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        
        BEGIN {
            total = 0;
        }
        
        {
            i = tolower($0);
            if (freq[i] == 1) {
                total = total + 2;
            } else if (freq[i] > 1) {
                total++;
            }
                freq[i]++;
        }
        
        END {
            print "Found", total, "lines with duplicate records.";
        }
      2. Execute the nDuplicates.awk file and pass the file to itself as input:

        awk -f nDuplicates.awk nDuplicates.awk
        
      3. The output will look similar to the following:

          
        Found 5 lines with duplicate records.
            
        
      4. Execute the command again, passing the file twice to itself:

        awk -f nDuplicates.awk nDuplicates.awk nDuplicates.awk
        
      5. The output will look similar to the following:

          
        Found 42 lines with records that already existed
        
        

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How To Install Python 3 and Set Up a Programming Environment on Debian 10


      Introduction

      Python is a flexible and versatile programming language suitable for many use cases, including scripting, automation, data analysis, machine learning, and back-end development. First published in 1991 with a name inspired by the British comedy group Monty Python, the development team wanted to make Python a language that was fun to use. Quick to set up with immediate feedback on errors, Python is a useful language to learn for beginners and experienced developers alike. Python 3 is the most current version of the language and is considered to be the future of Python.

      This tutorial will get your Debian 10 server set up with a Python 3 programming environment. Programming on a server has many advantages and supports collaboration across development projects.

      Prerequisites

      In order to complete this tutorial, you should have a non-root user with sudo privileges on a Debian 10 server. To learn how to achieve this setup, follow our Debian 10 initial server setup guide.

      If you’re not already familiar with a terminal environment, you may find the article “An Introduction to the Linux Terminal” useful for becoming better oriented with the terminal.

      With your server and user set up, you are ready to begin.

      Step 1 — Setting Up Python 3

      Debian Linux ships with both Python 3 and Python 2 pre-installed. To make sure that our versions are up-to-date, let’s update and upgrade the system with the apt command to work with the Advanced Packaging Tool:

      • sudo apt update
      • sudo apt -y upgrade

      The -y flag will confirm that we are agreeing for all items to be installed.

      Once the process is complete, we can check the version of Python 3 that is installed in the system by typing:

      You’ll receive output in the terminal window that will let you know the version number. While this number may vary, the output will be similar to this:

      Output

      Python 3.7.3

      To manage software packages for Python, let’s install pip, a tool that will install and manage programming packages we may want to use in our development projects. You can learn more about modules or packages that you can install with pip by reading “How To Import Modules in Python 3.”

      • sudo apt install -y python3-pip

      Python packages can be installed by typing:

      • pip3 install package_name

      Here, package_name can refer to any Python package or library, such as Django for web development or NumPy for scientific computing. So if you would like to install NumPy, you can do so with the command pip3 install numpy.

      There are a few more packages and development tools to install to ensure that we have a robust set-up for our programming environment:

      • sudo apt install build-essential libssl-dev libffi-dev python3-dev

      Once Python is set up, and pip and other tools are installed, we can set up a virtual environment for our development projects.

      Step 2 — Setting Up a Virtual Environment

      Virtual environments enable you to have an isolated space on your server for Python projects, ensuring that each of your projects can have its own set of dependencies that won’t disrupt any of your other projects.

      Setting up a programming environment provides us with greater control over our Python projects and over how different versions of packages are handled. This is especially important when working with third-party packages.

      You can set up as many Python programming environments as you want. Each environment is basically a directory or folder on your server that has a few scripts in it to make it act as an environment.

      While there are a few ways to achieve a programming environment in Python, we’ll be using the venv module here, which is part of the standard Python 3 library. Let’s install venv by typing:

      • sudo apt install -y python3-venv

      With this installed, we are ready to create environments. Let’s either choose which directory we would like to put our Python programming environments in, or create a new directory with mkdir, as in:

      • mkdir environments
      • cd environments

      Once you are in the directory where you would like the environments to live, you can create an environment by running the following command:

      Essentially, pyvenv sets up a new directory that contains a few items which we can view with the ls command:

      Output

      bin include lib lib64 pyvenv.cfg share

      Together, these files work to make sure that your projects are isolated from the broader context of your local machine, so that system files and project files don’t mix. This is good practice for version control and to ensure that each of your projects has access to the particular packages that it needs. Python Wheels, a built-package format for Python that can speed up your software production by reducing the number of times you need to compile, will be in the Ubuntu 18.04 share directory.

      To use this environment, you need to activate it, which you can achieve by typing the following command that calls the activate script:

      • source my_env/bin/activate

      Your command prompt will now be prefixed with the name of your environment, in this case it is called my_env. Depending on what version of Debian Linux you are running, your prefix may appear somewhat differently, but the name of your environment in parentheses should be the first thing you see on your line:

      This prefix lets us know that the environment my_env is currently active, meaning that when we create programs here they will use only this particular environment’s settings and packages.

      Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

      After following these steps, your virtual environment is ready to use.

      Step 3 — Creating a “Hello, World” Program

      Now that we have our virtual environment set up, let’s create a traditional “Hello, World!” program. This will let us test our environment and provides us with the opportunity to become more familiar with Python if we aren’t already.

      To do this, we’ll open up a command-line text editor such as nano and create a new file:

      Once the text file opens up in the terminal window we’ll type out our program:

      print("Hello, World!")
      

      Exit nano by typing the CTRL and X keys, and when prompted to save the file press y.

      Once you exit out of nano and return to your shell, let’s run the program:

      The hello.py program that you just created should cause your terminal to produce the following output:

      Output

      Hello, World!

      To leave the environment, simply type the command deactivate and you will return to your original directory.

      Conclusion

      Congratulations! At this point you have a Python 3 programming environment set up on your Debian 10 Linux server and you can now begin a coding project!

      If you are using a local machine rather than a server, refer to the tutorial that is relevant to your operating system in our “How To Install and Set Up a Local Programming Environment for Python 3” series.

      With your server ready for software development, you can continue to learn more about coding in Python by reading our free How To Code in Python 3 eBook, or consulting our Programming Project tutorials.

      Download our free Python eBook!

      How To Code in Python eBook in EPUB format

      How To Code in Python eBook in PDF format



      Source link

      How To Install F# and Set Up a Local Programming Environment on Ubuntu 18.04


      The author selected the Free Software Foundation to receive a donation as part of the Write for DOnations program.

      Introduction

      F# is an open-source programming language initially developed at Microsoft Research to extend .NET, Microsoft’s set of tools, libraries, and languages to build applications and services. Besides its remarkably concise syntax, F# supports multiple paradigms, meaning that it can do different types of code structuring, though it was primarily designed to take advantage of the functional programming approach.

      Adopting a specific paradigm, or a style of code, determines the way we will think and organize our programming problem solving. With an imperative approach, the design model used in languages like C++ or Java, a developer describes step-by-step how the computer must accomplish a task. It’s about writing a sequence of statements that will change memory states at the program’s execution. This works fine until we encounter some irregular situations. Consider a shared object for instance, which is used by multiple applications simultaneously. We might want to read its value at the same time that another component is modifying it. These are concurrent actions upon a memory location that can produce data inconsistency and undefined behavior.

      In functional code design, we prevent this kind of problem by minimizing the use of mutable states, or states that can change after we make them. Function is the keyword here, referring to mathematical transformations on some information provided as arguments. A functional code expresses what the program is by composing the solution as a set of functions to be executed. Typically, we build up layers of logic using functions that can return another function or take other functions as inputs.

      Functional programming with F# brings a number of benefits:

      • A more readable and expressive syntax that increases program maintainability.
      • A code less prone to breaking and easier to debug because of stateless functions that can be isolated for testing.
      • Native constructs that facilitate asynchronous programming and safer concurrency.
      • Access to all the existing tools in the .NET world including the community-shared packages.

      Choosing a Runtime

      Since F# is cross-platform, maintaining a similar execution model behavior through different operating systems is essential. .NET achieves this by means of a runtime. A runtime system is a piece of software that orchestrates the execution of a program written with a specific programming language, handling interfacing with the operating system and memory management, among other things.

      There are actually two .NET runtime implementations available on Linux: .NET Core and Mono. Historically, .NET only worked on Windows. In those days, one could resort to the community Mono project to run .NET applications on other platforms like Linux and macOS. Microsoft then launched .NET Core, a faster, modular subset of the original .NET framework, to target multiple platforms.

      At the time of this tutorial’s publication, they both can be used for building web applications or command line utilities. That said, .NET Core does not ship models to create GUI desktop applications on Linux and macOS, while Mono is the only one to support mobile and gaming platforms. It is important to know these differences since the runtime you pick will shape the programs you will build. You could also choose to have both .NET Core and Mono installed in order to account for all use cases and to make a more productive stack.

      In this tutorial, you will set up an F# programming environment on Ubuntu 18.04 using both .NET Core and Mono runtimes. You will then write some code examples to test and review build and compile methods.

      Prerequisites

      To complete this tutorial, you will need basic familiarity with the command line and a computer running Ubuntu 18.04 with a non-root user with sudo privileges.

      Step 1 — Installing F# with .NET Core

      Microsoft provides the .NET Core Software Development Kit (SDK) for F# developers. A Software Development Kit is a set of programming tools that allows programmers to produce specialized applications and adapt them to various operating systems. It traditionally includes a text editor, languages support, a runtime, and a compiler, among other components. In this step, you are going to install this SDK. But first, you will register the Microsoft repository and fetch some dependencies.

      You’ll be completing the installation and setup on the command line, which is a non-graphical way to interact with your computer. That is, instead of clicking on buttons, you’ll be typing in text and receiving feedback from your computer through text as well.

      The command line, also known as a shell or terminal, can help modify and automate many of the tasks you do on a computer every day, and is an essential tool for software developers. There are many terminal commands to learn that can enable you to do more powerful things. For more information about the command line, check out the Introduction to the Linux Terminal tutorial.

      On Ubuntu 18.04, you can find the Terminal application by clicking on the Ubuntu icon in the upper-left hand corner of your screen and typing terminal into the search bar. Click on the Terminal application icon to open it. Alternatively, you can hit the CTRL, ALT, and T keys on the keyboard at the same time to open the Terminal application automatically.

      Ubuntu Terminal

      Once you have opened the terminal, use the wget command to download a package containing some required files, the Microsoft repository configurations, and a key for server communication.

      • wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb

      Now, add the Microsoft repository and install the packages to your system using the dpkg -i instruction.

      • sudo dpkg -i packages-microsoft-prod.deb

      Next, activate the Universe repository, which on Ubuntu is a community-maintained archive of software that is free and open source. This will give you access to apt-transport-https, a dependency for enabling the Ubuntu package manager APT transport over HTTPS.

      • sudo add-apt-repository universe
      • sudo apt install apt-transport-https

      Next, update available downloads:

      Finally, install the current version of the .NET SDK. This tutorial will use version 2.2:

      • sudo apt install dotnet-sdk-2.2

      Now that you have the .NET SDK installed, a quick way to check if everything went well is to try the .NET Core command line interface (CLI), which will be available in the shell once the SDK is downloaded and installed. Display information about your .NET setup by typing this in your terminal:

      When you run a dotnet command for the first time, a text section is displayed as shown below:

      Output

      Welcome to .NET Core! --------------------- Learn more about .NET Core: https://aka.ms/dotnet-docs Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs Telemetry --------- The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous and doesn't include command-line arguments. The data is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell. Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry ...

      This notification is about collected data, and explains that some .NET CLI commands will send usage information to Microsoft. You will disable this in a moment; for now, look at the output from dotnet --info.

      After a brief moment, the terminal will list information about your .NET installation:

      Output

      .NET Core SDK (reflecting any global.json): Version: 2.2.101 Commit: 236713b0b7 Runtime Environment: OS Name: ubuntu OS Version: 18.04 OS Platform: Linux RID: ubuntu.18.04-x64 Base Path: /usr/share/dotnet/sdk/2.2.101/ Host (useful for support): Version: 2.2.0 Commit: 1249f08fed .NET Core SDKs installed: 2.2.101 [/usr/share/dotnet/sdk] .NET Core runtimes installed: Microsoft.AspNetCore.All 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App] To install additional .NET Core runtimes or SDKs: https://aka.ms/dotnet-download

      Depending on the SDK version, the output may be slightly different, but this confirms that .NET Core is ready to use.

      As mentioned before, the telemetry feature allows some .NET CLI commands to send usage information to Microsoft. It is enabled by default, and can be deactivated by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1. To do so, add a new line to your .profile environment customization file by opening it in your text editor. For this tutorial, we will use nano:

      Add the following line to the end of .profile:

      ~/.profile

      . . .
      export DOTNET_CLI_TELEMETRY_OPTOUT=1
      

      Exit nano by pressing the CTRL and X keys. When prompted to save the file, press Y and then ENTER.

      You can activate the new configuration using the source command:

      From now on, telemetry will be turned off at startup.

      At this point you have .NET Core runtime, languages support, and libraries installed, allowing you to run and build some .NET applications. The dotnet CLI is also available for managing .NET source code and binaries. You could start building F# projects, but as mentioned previously, the .NET Core environment does not provide all the constructs needed to be completely cross-platform. For now you cannot use it to develop mobile applications, for example.

      In order to solve this problem, in the next step you will install F# again, but this time with Mono.

      Step 2 — Installing F# with Mono

      You can use Mono to fill in the remaining gaps in capability left by .NET Core. Mono and .NET Core are both based on the same standard library and both support .NET languages, but that is where the similarity ends. They use different runtimes, different CLIs, and different compilers, making it possible for them to be installed side by side to create a more reliable programming environment. In this section you are going to supplement your environment with the Mono tools for .NET programming and run an F# program from the command line.

      A version of Mono is available in the Ubuntu repositories, but this can be outdated. Instead, add the official Mono package repository to your package manager:

      • sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
      • echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list

      In the preceding commands, you used apt-key to retrieve keys for securing packages transferred from the official Mono repositories. You then added the Mono packages source to your repositories list.

      With a new source list added for APT, update your repositories:

      Next, download the Mono tools. Unlike .NET Core, Mono does not include F# tools, so you will download it as a separate package. Install fsharp and the mono-complete meta-package using the following command:

      • sudo apt install mono-complete fsharp

      Note: Because of the size of this download, the installation process for mono-complete may take a while.

      Once done, you will have the compiler fsharpc and an interactive shell called fsharpi or simply FSI. FSI is an environment, inside the shell, that receives user's input as an expression, evaluates it, then outputs the result and waits for another input. It is just like typing a command in the traditional shell and seeing the result, except here, inputs are F# expressions. FSI provides a fast method to test code or run scripts.

      Activate FSI with the following command:

      This will start the interactive session and replace your regular prompt with the fsharpi prompt:

      Output

      Microsoft (R) F# Interactive version 4.1 Copyright (c) Microsoft Corporation. All Rights Reserved. For help type #help;; >

      You can return to the default shell by running #quit;;. In fsharpi, each command line ends with a double semicolon.

      Let's try a simple operation using the printfn function to render a message passed as a parameter:

      You will receive the following output:

      Output

      Hello World! val it : unit = () >

      From the preceding interaction, fsharpi evaluates the expression as a unit type value. The code is then executed and the result is printed with its type.

      fsharpi can also run a file containing F# code. The script must be named with a .fsx extension and executed from the shell with the command:

      Now that you know the F# installation is working, leave the shell with:

      > #quit;;
      

      With Mono and .NET Core installed, you are now prepared to write any type of F# programs. FSI will allow you to test your code and run some scripts if needed, but executions will be slow. For your F# script to be executed, additional steps are performed to translate the source code into artifacts understandable by the processor, hence the slowness. To remedy this, in the next section you will compile your code with .NET Core, creating standalone binary files that can be immediately run by the machine.

      Step 3 — Writing and Compiling F# Programs with .NET Core

      In this step, you will compile F# source code via command line compilers provided with .NET Core. This will allow you to make your applications faster and to produce preset executable packages for specific systems, making your program easier to distribute.

      Compiling is the transformation process that turns source code into binary file. The software that accomplishes this conversion is called a compiler. .NET Core relies on the dotnet CLI to perform compiling. To demonstrate this, you are going to create a basic F# source to review the compilation cases.

      The dotnet CLI provides a complete application build toolchain. In general, an association of a command and the dotnet driver is used in the shell to complete a task. For example:

      • dotnet new will create a project
      • dotnet build will build a project and all of its dependencies
      • dotnet add package will add a package reference to a project file

      The following will create a new console project called FSharpHello. The -lang option sets the programming language you will code with while the -o option creates a directory in which to place the output.

      • dotnet new console -lang F# -o FSharpHello

      Once this is done, navigate into your newly created project directory:

      This directory contains the FSharpHello.fsproj project configuration file and the obj folder which is used to store temporary object files. There is also the Program.fs file where your default source code exists. Open it in your text editor:

      The file has been automatically filled with a Hello World program:

      Program.fs

      // Learn more about F# at http://fsharp.org
      
      open System
      
      [<EntryPoint>]
      let main argv =
          printfn "Hello World from F#!"
          0 // return an integer exit code
      

      In this code, you start importing the System module with open System, then you define the program entry point, i.e., the place where the program starts when launched from the shell. The main function will call for a Hello World message printing to the console and will stop the program (return an integer exit code).

      Exit out of the file.

      To compile and run this code, use the following from the project directory ~/FSharpHello:

      The program will run, printing the following output to the screen:

      Output

      Hello World from F#!

      Note that it took a while for this program to run, just as with the FSI. As we mentioned before, it's possible to run this faster by generating an executable, i.e., a binary file that can be run directly by the operating system. Here is how to achieve this:

      • dotnet publish -c release -r linux-x64

      This will produce the executable bin/release/netcoreapp2.2/linux-x64/publish/FSharpHello.dll file. This is a shared library that will run on a 64-bit Linux architecture. To export a generic executable for macOS systems, you would replace the linux-x64 runtime identifier (RID) with osx-x64.

      Now execute the file with the following command:

      • dotnet bin/release/netcoreapp2.2/linux-x64/publish/FSharpHello.dll

      This time, you will receive the output much quicker, since the program is already translated into binary.

      Now that you know how to compile in .NET Core, let's see how Mono compiles programs with the dedicated fsharpc command.

      Step 4 — Writing and Compiling F# Programs with Mono

      Mono's compilation process is similar to that of .NET Core, but this time there is a specific command used to compile the program. The fsharpc command is the tool, and it has been created only for compiling.

      This time, create a hello.fs file and write some F# code. First, return to your home directory:

      Next, open up a new file named hello.fs:

      Add the following line to the file:

      hello.fs

      open System
      

      As seen before, this imports the System module or namespace, giving you access to built-in system functions and objects like Console.

      Now, add in some more lines of code:

      hello.fs

      open System
      
      let hello() =
          printf "Who are you? "
          let name = Console.ReadLine()
          printfn "Oh, Hello %s!nI'm F#." name
      

      These new lines define the hello() function to read user input and print a feedback message.

      Now you can add the final lines:

      hello.fs

      open System
      
      let hello() =
          printf "Who are you? "
          let name = Console.ReadLine()
          printfn "Oh, Hello %s!nI'm F#." name
      
      hello()
      Console.ReadKey() |> ignore
      

      Here you are calling the function hello(), then using the ReadKey() method to end the program with a final keystroke.

      Save and exit the file.

      Now with the fsharpc command, use the -o flag to define the output filename and compile your hello.fs source code like this:

      • fsharpc hello.fs -o hello

      The preceding command will generate a hello executable file you can run with the mono command:

      This gives you the following output and awaits user input:

      Output

      Who are you?

      If you type in Sammy, you will get the following.

      Output

      Oh, Hello Sammy! I'm F#.

      Press a final keystroke, and the program will end.

      Congratulations! You have written and compiled your first F# program, both with Mono and .NET Core.

      Conclusion

      In this tutorial, you installed tooling for F# programming, covering both .NET Core and Mono environments. You also tested examples of F# code and built executables. These are the first steps toward learning this practical functional language.

      Next steps could be to learn the language and get in touch with the community. Also, with projects getting more complex, you might need to manage code and resources more efficiently. Package managers like NuGet or Paket are bridges to the strong ecosystem built around .NET and tools-of-choice for organizing large programs.



      Source link