One place for hosting & domains

      External

      How To Use subprocess to Run External Programs in Python 3


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Python 3 includes the subprocess module for running external programs and reading their outputs in your Python code.

      You might find subprocess useful if you want to use another program on your computer from within your Python code. For example, you might want to invoke git from within your Python code to retrieve files in your project that are tracked in git version control. Since any program you can access on your computer can be controlled by subprocess, the examples shown here will be applicable to any external program you might want to invoke from your Python code.

      subprocess includes several classes and functions, but in this tutorial we’ll cover one of subprocess’s most useful functions: subprocess.run. We’ll review its different uses and main keyword arguments.

      Prerequisites

      To get the most out of this tutorial, it is recommended to have some familiarity with programming in Python 3. You can review these tutorials for the necessary background information:

      Running an External Program

      You can use the subprocess.run function to run an external program from your Python code. First, though, you need to import the subprocess and sys modules into your program:

      import subprocess
      import sys
      
      result = subprocess.run([sys.executable, "-c", "print('ocean')"])
      

      If you run this, you will receive output like the following:

      Output

      ocean

      Let’s review this example:

      • sys.executable is the absolute path to the Python executable that your program was originally invoked with. For example, sys.executable might be a path like /usr/local/bin/python.
      • subprocess.run is given a list of strings consisting of the components of the command we are trying to run. Since the first string we pass is sys.executable, we are instructing subprocess.run to execute a new Python program.
      • The -c component is a python command line option that allows you to pass a string with an entire Python program to execute. In our case, we pass a program that prints the string ocean.

      You can think of each entry in the list that we pass to subprocess.run as being separated by a space. For example, [sys.executable, "-c", "print('ocean')"] translates roughly to /usr/local/bin/python -c "print('ocean')". Note that subprocess automatically quotes the components of the command before trying to run them on the underlying operating system so that, for example, you can pass a filename that has spaces in it.

      Warning: Never pass untrusted input to subprocess.run. Since subprocess.run has the ability to perform arbitrary commands on your computer, malicious actors can use it to manipulate your computer in unexpected ways.

      Capturing Output From an External Program

      Now that we can invoke an external program using subprocess.run, let’s see how we can capture output from that program. For example, this process could be useful if we wanted to use git ls-files to output all your files currently stored under version control.

      Note: The examples shown in this section require Python 3.7 or higher. In particular, the capture_output and text keyword arguments were added in Python 3.7 when it was released in June 2018.

      Let’s add to our previous example:

      import subprocess
      import sys
      
      result = subprocess.run(
          [sys.executable, "-c", "print('ocean')"], capture_output=True, text=True
      )
      print("stdout:", result.stdout)
      print("stderr:", result.stderr)
      

      If we run this code, we’ll receive output like the following:

      Output

      stdout: ocean stderr:

      This example is largely the same as the one introduced in the first section: we are still running a subprocess to print ocean. Importantly, however, we pass the capture_output=True and text=True keyword arguments to subprocess.run.

      subprocess.run returns a subprocess.CompletedProcess object that is bound to result. The subprocess.CompletedProcess object includes details about the external program’s exit code and its output. capture_output=True ensures that result.stdout and result.stderr are filled in with the corresponding output from the external program. By default, result.stdout and result.stderr are bound as bytes, but the text=True keyword argument instructs Python to instead decode the bytes into strings.

      In the output section, stdout is ocean (plus the trailing newline that print adds implicitly), and we have no stderr.

      Let’s try an example that produces a non-empty value for stderr:

      import subprocess
      import sys
      
      result = subprocess.run(
          [sys.executable, "-c", "raise ValueError('oops')"], capture_output=True, text=True
      )
      print("stdout:", result.stdout)
      print("stderr:", result.stderr)
      

      If we run this code, we receive output like the following:

      Output

      stdout: stderr: Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: oops

      This code runs a Python subprocess that immediately raises a ValueError. When we inspect the final result, we see nothing in stdout and a Traceback of our ValueError in stderr. This is because by default Python writes the Traceback of the unhandled exception to stderr.

      Raising an Exception on a Bad Exit Code

      Sometimes it’s useful to raise an exception if a program we run exits with a bad exit code. Programs that exit with a zero code are considered successful, but programs that exit with a non-zero code are considered to have encountered an error. As an example, this pattern could be useful if we wanted to raise an exception in the event that we run git ls-files in a directory that wasn’t actually a git repository.

      We can use the check=True keyword argument to subprocess.run to have an exception raised if the external program returns a non-zero exit code:

      import subprocess
      import sys
      
      result = subprocess.run([sys.executable, "-c", "raise ValueError('oops')"], check=True)
      

      If we run this code, we receive output like the following:

      Output

      Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: oops Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-c', "raise ValueError('oops')"]' returned non-zero exit status 1.

      This output shows that we ran a subprocess that raised an error, which is printed in stderr in our terminal. Then subprocess.run dutifully raised a subprocess.CalledProcessError on our behalf in our main Python program.

      Alternatively, the subprocess module also includes the subprocess.CompletedProcess.check_returncode method, which we can invoke for similar effect:

      import subprocess
      import sys
      
      result = subprocess.run([sys.executable, "-c", "raise ValueError('oops')"])
      result.check_returncode()
      

      If we run this code, we’ll receive:

      Output

      Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: oops Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/subprocess.py", line 444, in check_returncode raise CalledProcessError(self.returncode, self.args, self.stdout, subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-c', "raise ValueError('oops')"]' returned non-zero exit status 1.

      Since we didn’t pass check=True to subprocess.run, we successfully bound a subprocess.CompletedProcess instance to result even though our program exited with a non-zero code. Calling result.check_returncode(), however, raises a subprocess.CalledProcessError because it detects the completed process exited with a bad code.

      Using timeout to Exit Programs Early

      subprocess.run includes the timeout argument to allow you to stop an external program if it is taking too long to execute:

      import subprocess
      import sys
      
      result = subprocess.run([sys.executable, "-c", "import time; time.sleep(2)"], timeout=1)
      

      If we run this code, we’ll receive output like the following:

      Output

      Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/subprocess.py", line 491, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/local/lib/python3.8/subprocess.py", line 1024, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/local/lib/python3.8/subprocess.py", line 1892, in _communicate self.wait(timeout=self._remaining_time(endtime)) File "/usr/local/lib/python3.8/subprocess.py", line 1079, in wait return self._wait(timeout=timeout) File "/usr/local/lib/python3.8/subprocess.py", line 1796, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['/usr/local/bin/python', '-c', 'import time; time.sleep(2)']' timed out after 0.9997982999999522 seconds

      The subprocess we tried to run used the time.sleep function to sleep for 2 seconds. However, we passed the timeout=1 keyword argument to subprocess.run to time out our subprocess after 1 second. This explains why our call to subprocess.run ultimately raised a subprocess.TimeoutExpired exception.

      Note that the timeout keyword argument to subprocess.run is approximate. Python will make a best effort to kill the subprocess after the timeout number of seconds, but it won’t necessarily be exact.

      Passing Input to Programs

      Sometimes programs expect input to be passed to them via stdin.

      The input keyword argument to subprocess.run allows you to pass data to the stdin of the subprocess. For example:

      import subprocess
      import sys
      
      result = subprocess.run(
          [sys.executable, "-c", "import sys; print(sys.stdin.read())"], input=b"underwater"
      )
      

      We’ll receive output like the following after running this code:

      Output

      underwater

      In this case, we passed the bytes underwater to input. Our target subprocess used sys.stdin to read the passed in stdin (underwater) and printed it out in our output.

      The input keyword argument can be useful if you want to chain multiple subprocess.run calls together passing the output of one program as the input to another.

      Conclusion

      The subprocess module is a powerful part of the Python standard library that lets you run external programs and inspect their outputs easily. In this tutorial, you have learned to use subprocess.run to control external programs, pass input to them, parse their output, and check their return codes.

      The subprocess module exposes additional classes and utilities that we did not cover in this tutorial. Now that you have a baseline, you can use the subprocess module’s documentation to learn more about other available classes and utilities.



      Source link

      How to Configure Nextcloud to use Linode Object Storage as an External Storage Mount


      Updated by Linode

      Contributed by
      Linode

      Marquee image for How to Configure Nextcloud to use Linode Object Storage as an External Storage Mount

      Nextcloud is an open source solution for file hosting and sharing. With Nextcloud, you can synchronize files from a local computer to the Linode server and share them with collaborators. Nextcloud’s customizable security features and intuitive user interface keeps the files safe and easy to manage.

      You can configure Nextcloud to enable external storage devices and services, like Linode Object Storage, to use as a secondary place to store files. Using Linode Object Storage to store files prevents you from running out of storage space limited by the Linode’s plan size. When using Nextcloud’s graphical user interface (GUI) to manage files, the external storage device shows up just like any other folder.

      Before You Begin

      1. Deploy a Nextcloud server instance. You can use the Linode Nextcloud One-Click App for an easy and quick deployment.

      2. Enable the Object Storage service on your Linode account.

      3. Generate Object Storage access keys.

      4. If you are not familiar with Linode Object Storage, review the How to Use Linode Object Storage guide.

      In This Guide

      Nextcloud Configurations

      Enable the External Storage App

      In this section you enable the External Storage Support Nextcloud app in order to use external storage sources.

      Note

      You must belong to the admin user group in order to install the External storage support app.

      1. Log into your Nextcloud instance.

      2. Click the user icon (or cog wheel) in the top navigation menu and select Apps.

        Access Nextcloud App settings.

      3. Click Files to access all Nextcloud apps related to file management, in the left-hand navigation menu.

        Access the Files App settings.

      4. Use the search field in the top navigation to narrow down the visible apps. You can enter external as your search term.

      5. Viewing the External storage support app, click the Enable button in order to install it to the Nextcloud instance.

        Enable the external storage support app.

      Create a New Linode Object Storage External Storage Mount

      After enabling the External Storage Support app, you are now ready to add a new external storage mount. You configure the new external storage mount to use the Linode Object Storage service.

      Note

      1. Click the user icon (or cog wheel) in the top navigation menu and select Settings.

        Access Nextcloud settings.

      2. In the left-hand navigation menu, under the Administration heading, click External Storages. The External Storages administration page appears.

        Access external storage configurations.

      3. In the Folder name text entry box, provide a name for the external storage directory.

      4. From the External Storage dropdown menu, select the Amazon S3 option.

        Note

        Linode Object Storage is S3-compatible. Nextcloud connects to Amazon’s Object Storage service by default, however, in the next step you override the default behavior to use Linode Object Storage hosts instead.

      5. Select Access Key from the Authentication dropdown menu.

      6. Under the Configuration heading, provide the following configurations:

        Configuration Description
        Bucket The name to assign to the Object Storage bucket. If this bucket name already exists in the data center region you select, an error occurs.
        Hostname The hostname used for the Object Storage region where the bucket is be stored. Refer to the Linode Object Storage Region and Hostname Values note located below this table for available hostname values.
        Port The port number to use to access the Object Storage host. This value must be 443.
        Region The data center region to store your Object Storage bucket. Refer to the Linode Object Storage Region and Hostname Values note located below this table for available data center region IDs.
        Enable SSL A configuration to enable secure sockets layer (SSL). This configuration must be enabled.
        Enable Path Style This configuration changes the default path format used by Nextcloud to access the Object Storage bucket. Do not enable this configuration.
        Legacy (v2) Authentication This configuration enables version 2 authentication to the Object Storage service. By default Nextcloud uses version 4 authentication that is compatible with Linode Object Storage. Do not enable this configuration.
        Access Key The value of the Access Key you created using the Linode Cloud Manager.
        Secret Key The value of the Secret Key you created using the Linode Cloud Manager.



        Linode Object Storage Region and Hostname Values

        Region Region ID Hostname
        Newark, NJ, USA us-east-1 us-east-1.linodeobjects.com
        Frankfurt, Germany eu-central-1 eu-central-1.linodeobjects.com
        Singapore, Singapore ap-south-1 ap-south-1.linodeobjects.com
      7. In the Available for text entry box, enter the group name(s) you would like to give access to the Linode Object Storage external storage. To learn more about user and group permissions related to external storage, see Nextcloud’s documentation.

      8. Click the check icon to save the configurations. If all your configurations are valid, you should see a green check box appear next to the external storage entry.

        Save your external storage configurations.

      9. Using the top navigation menu, click the Files menu item. The external storage folder appears in the list of folders.

        Access all your Nextcloud files.

      10. Click the external storage folder to view its contents. You should not see anything stored there yet.

      11. Test out the external storage mount by adding a file to the folder. Click the + button in the top breadcrumbs area of the screen and select Upload file.

        Upload a file to your Linode Object Storage bucket.

      12. The local file browser appears. Select a test file to add to the external storage folder and click Open. The file appears in the folder.

        Your should see your uploaded file appear in the folder.

        Note

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      This guide is published under a CC BY-ND 4.0 license.



      Source link