GIS Python Workflows While You Sleep: An Intro To Automation
/
Sometimes the GIS workflows you need to conduct can be tedious and repetitive, or your workflows cannot stop just because the workday is over. It may be beneficial to perform a task when network latency is low, and users do not lock data. There are more tools than ever to automate GIS processes.
Using the Python scripting language, Esri’s ArcPy site package, and the new ArcGIS API for Python, you can reduce hours spent and remove human error automating GIS tasks. Combined with the Task Scheduler application native to the Windows operating system, these tools allow you to run the tasks on a schedule.
Let’s dig deeper into how to set up a basic GIS automation task to enable you to run a Python script on a regular schedule.
Script Preparation
Setting the Stage
The first thing needed is a Python script and a computer with the appropriate version of Python installed. It is important to cover some Python setup best practices for ensuring your script is successful when it runs. My recommendations include:
Do not collect user input.
Use ArcGIS API for Python for web data.
Ensure proper error handling is in place.
When you are fully automating a script to run, usually after business hours, no one will be around to provide input to your script when it runs. The process is initiated even if your computer is locked. Avoid getting Python parameters using arcpy.GetParameter() or arcpy.GetParameterAsText() as these functions require an input from a script tool parameter in the desktop GIS environment. This process is designed to work without user input. The paths and data parameters will need to be hardcoded unless a unique process can pull the parameters from another location.
Using the ArcGIS API for Python
If the process is running locally in a file geodatabase or Enterprise geodatabase, you should be able to conduct your workflows using ArcPy. If you are working with ArcGIS Online (AGOL) or Portal for ArcGIS, you should consider using the ArcGIS API for Python to work with your layers. Older ArcPy modules created for working with web data are out of date and not supported anymore, making them more challenging to use. The API for Python is straight forward and intuitive to use with web layers and maps. You will need to have the API for Python installed with Python 3 on the machine running the script. If you have the most recent version of ArcGIS Pro desktop software, these are installed already. Although the Pro software is not required to be running when the script starts, it is important to be logged into using appropriate credentials to access the necessary licenses to run your script.
Tracking the Good and the Bad
Despite your best efforts setting up the script, there are always problems that may be encountered. Problems can include loss of access to a data source, credentials becoming invalid, license expiration, and many other potential issues. Error handling techniques are required to deal with these issues. Beyond the standard Try/Except statements within your code, it is recommended that you also build in the ability to either save a log file, email an admin, or text someone that an issue occurred. At a minimum, creating a log file to save a record of what happened each time the script ran will give you extra tools for determining what went wrong with the script or keep track of what is going right.
Task Creation
Once you have a working script that has been tested and debugged with error handling in place, it is time to create the scheduled task in your operating system. First, you need to identify the path to Python on your machine. This can be done using your preferred Integrated Development Environment (IDE). For this demonstration, we will use Spyder since it can be installed as a plug-in from ArcGIS Pro.
Identifying the Python Path
Create a script with the following code and run it as you would typically run a script. Remember, this must be done using the correct version of Python. If you are using ArcGIS Pro or the ArcGIS API for Python, you will need to use Python 3.
The results of running this script give you information about where Python is located on your computer. The Python path is used as a parameter in the Task Scheduler set up.
Creating the Task
This workflow uses Windows 10. Open the start menu and search for Task Scheduler. Open the application. On the right-hand side of the application, in the Actions window, click “Create Task…”.
General Tab
The window that opens has tabs along the top. Several of them require parameters from you to set up correctly. The General tab requires a name for the task and a description if you choose to add one. The user account must be an admin on the computer being used. It is recommended that the task be set up to run whether the user is logged in or not. That way, the script can run after hours when no one is using the computer. The computer must be on, but the admin user does not have to be logged in.
The option to store the password is up to you. If the script requires access to resources outside of the local computer, you must save the password. Otherwise, leave it unchecked. You might have to experiment with the setting “Run with highest privileges” as it depends on what task your script is trying to complete.
Trigger Tab
The Trigger tab determines when the task will be run. The run's timing is entirely dependent on your particular needs and can be set to run as often as you like. Again, if it is a resource-intensive task, you may want to schedule the task to run when those resources are available or are not being heavily used.
Actions Tab
The most important tab is the Actions tab. Here we define where Python is locally installed and the path to the script you wish to run. The Python path was identified in the previous step. There may be several Python locations on your machine, and it depends on the software installed or how you set up Python. Here, we will assume that a standard installation set up of ArcGIS Pro has been done. If you run the script above, you might get back the following path for Python. This path is a common location for Python 3 on a machine that has ArcGIS Pro currently installed.
C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Python.exe
Esri also recommends using propy.bat, installed with the Pro software, as a reliable method of activating the correct Python version on your computer.
C:\Program Files\ArcGIS\Pro\bin\Python\scripts\propy.bat
If you have cloned your Python environment or modified it in any way, you would need to find and use the correct path. Regardless of the Python path you use, it needs to be added to the Program/Script parameter of the task when you create a new action on the Actions tab. On the Actions tab, click New at the bottom of the screen to see this menu. The path to the custom script you created goes in the Add Arguments field. It is recommended that you encase these paths in double quotes to avoid spaces or other path issues. Click OK once you have added these parameters. The Conditions and Settings tabs have settings that pertain to when and how the script runs, but no changes from the defaults are necessary. You can adjust these as needed.
Testing and Running
After the setup is complete, run the Python script several times to ensure it is functioning correctly. This can be done by manually triggering the script to run. On the main screen of the Task Scheduler window, click on Task Scheduler Library on the screen's left-hand side. Find your new task in the list of tasks. Right-click and select Run to start the task. It will say “This operation completed successfully” in the Last Run Result column if it was successful. Check the data to see if your workflow was successful. If it worked, set the task to run 10 minutes in the future and wait to see if it triggers automatically at that time. If it does, then you are good to go. Set up the task to run at your designated time and check the results after each run for several days to ensure it is still working properly. As mentioned previously, use logging methods to save the script results in a text document that you can check each day. For scripts that are scheduled to run infrequently, such as once a month, set it to run every day for a week to monitor its progress and correct any encountered errors.
If you wish to use the task settings on another computer, right-click the Task Scheduler task, and select Export to save it as an XML format file. This can then be imported into another computer’s Task Scheduler. It will need some updating potentially for the paths and the Admin user, but it will save time if you have to implement a similar task to run on multiple computers.
