Python Scripting

Goal and Objective

The goal of these labs was to introduce python scripting functions, learn how it can be utilized in ArcGIS, and gain some knowledge in how python works.

Background

Python scripting is an invaluable tool in the ArcGIS world if you want to perform higher ability functions in the program. Every tool present in the ArcToolbox is scripted using python code and can be drawn upon for use in the python script within Arc. Our labs goal was to learn and perform python scripting for several purposes. Python can also be used across several other softwares as an acceptable form of scripting language.

Scripts

The first script that our class learned was how to add xy coordinate to ArcGIS, convert the coordinate system to UTM_Zone_16N, and then to convert the map units. This is all present in the script below:

#Cody Kroening
#GEOG 337

#Import arcpy
import.arcpy

#Input feature class from GCS_WGS_1984 to UTM_NAD_1983_ZONE_16N

# Replaces a layer or table name with a path to a dataset or create the layer or table view within the script
arcpy.Project_management("XYdata","W:/geog/CHupy/geog337_f13/KROENICJ/EX1.gbd/Script1.py", "PROJCS['NAD_1983_UTM_Zone_16N',GEOGCS['GCS_North_American_1983',DATUM['D_North_American_1983', SPHEROID['GRS_1980',6378137.0,298.257222101]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Transverse_Mercator'],PARAMETER['False_Easting',500000.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',-87.0],PARAMETER['Scale_Factor',0.9996],PARAMETER['Latitude_Of_Origin',0.0],UNIT['Meter',1.0]]","WGS_1984_(ITRF00)_To_NAD_1983","GEOGCS['GCS_WGS_1984', DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]]")

#Adds XY coordinates to the input feature class
arcpy.AddXY_management("XYdataProject1")

The second script that I created was tasked with creating buffers that followed airport parameters as seen below. It also creates a distance buffer from the airport.

#Cody Kroening

#Geog 337: GIS 2

# Import arcpy module

import arcpy

# Local variables:

airports_shp = "W:\\geog\\CHupy\\geog337_f13\\KROENICJ\\Ex7\\Results\\airports.shp"

seaplaneairports_shp = "W:\\geog\\CHupy\\geog337_f13\\KROENICJ\\Ex7\\Results\\seaplaneairports.shp"

seabasebuffer_shp = "W:\\geog\\CHupy\\geog337_f13\\KROENICJ\\Ex7\\Results\\seabasebuffer.shp"

airportsselect_shp = "W:\\geog\\CHupy\\geog337_f13\\KROENICJ\\Ex7\\Results\\airportsselect.shp"

airportsbuffer_shp = "W:\\geog\\CHupy\\geog337_f13\\KROENICJ\\Ex7\\Results\\airportsbuffer.shp"

# Process: Select

arcpy.Select_analysis(airports_shp, seaplaneairports_shp, "\"FEATURE\" = 'Seaplane Base'")

# Process: Buffer

arcpy.Buffer_analysis(seaplaneairports_shp, seabasebuffer_shp, "7500 Meters", "FULL", "ROUND", "NONE", "")

# Process: Select (2)

arcpy.Select_analysis(airports_shp, airportsselect_shp, "\"FEATURE\" = 'Airport'")

# Process: Buffer (2)

arcpy.Buffer_analysis(airportsselect_shp, airportsbuffer_shp, "15000 Meters", "FULL", "ROUND", "NONE", "")

Script 3: The final script below was created to print stream names and along with the type of feature class.

#Cody Kroening

#Geog 337: GIS 2

# Import arcpy module

import arcpy

#Print stream name and feature class type

from arcpy import env

env.workspace="W:/geog/CHupy/geog337_f13/KROENICJ/Ex6/Results/NM.gdb"fclist=arcpy.ListFeatureClasses("*")

for fc in fclist:

fcdescribe = arcpy.Describe(fc)

print fcdescribe.name + " is a " + fcdescribe.dataType

Conclusion

Python scripting is a very valuable tool in order to peform higher level functioning and abilities in ArcGIS. It allows for the processing of multiple tools at one as long as you understand the scripting and its capabilities. Python not only allows you to create your own tools and to publish them, but is designed into the tools that we use everyday in the GIS community. It also allows users to create paramters for tools where a generic tool may not reach your specific needs. In conclusion, python scripting is an essential language to learn in higher level geospatial analysis in order for the user to be ahead of the game.

Note: I don't know how to revert the background color of the text to normal, so I'm sorry for the odd configuration.

Sources

Zandbergen, Paul A. January 9, 2013. "Python Scripting for ArcGIS." California: ESRI Press.

Raster Modeling and Sand Mine Suitability

Goals and Objectives

This latest and final extension into our investigation of sand mines in Trempealeau County contains raster modeling of both risk and suitability. The extension of spatial analysis had to be activated in order to work with raster models. All of the tools that were used to perform our analysis were located in the spatial analyst toolbox. Model builder was also utilized in order to model our workflow for both presentations sake and organisation. A suitability index, risk index, and viewshed were all created to best model this region in Trempealeau County. The goal of this project was to show areas of risk where the environment or population could be affected, places of great suitability for mining and to show regions where they are overlapping to show where low risk and high suitability match.

Data

Digital Elevation Model (DEM) - USGS data
Geology - Trempealeau County Class Folder
Landcover - National Land Cover Dataset (NLCD)
Streams - Trempealeau database
Railroads - Trempealeau Data from class
Populated Areas - ESRI folder
Prime Farmland - Trempealeau database
Schools - ESRI folder
Water Table Depth - Hydroline data in WGS

Methods

The tools that were used in this study were slope, reclassify, Raster Calculator, Euclidean Distance, Viewshed, Topo to Raster and Filter. Slope is a tool used to calculate a slopes percentage from elevation data in DEM. The reclassify tool was used to change the values field using class breaks. The reclassify tool allows easier performance of the raster calculator tool. The raster calculator tool allows for the junction of several rasters. It uses mathematical operators to perform further analysis. The Euclidean Distance tool that describes each pixel cells relationship to a source pixel. The viewshed tool represents the visible area that can be seen from a polyline/point in an elevation model. The topo to raster tool allows for the creation of a raster through many factors and in our case, contour lines. The final tool used was the filter tool, which either runs a low or high pass filter on a DEM to bring out or subdue features in the image.

Figure 1: Suitability Model

 

Figure 2: Risk Model

 



Figure 3: Viewshed Model and Risk/Suitability Combination Model

 

 

Results

After each model came to a conclusion 4 maps were the ending result. My first map, shown in figure 4, was created as suitability model using several factors that can be found in figure 1 above. The two geologic layers that we used to find the most prime sand came from the Wonewoc and Jordan formations. The second layer that I used to create my suitability model was NLCD data. When I performed the reclassify on my data, I chose barren land as the most suitable, then shrubland, herbaceous, and cultivated as the next most suitable. Finally forested land was the least suitable. In the reclassify function I also discluded land features that were wetlands, water, or developed regions.

The next step that I ran was the euclidean distance function from the railroad terminals. Then I reclassified my function which listed regions that are near a railroad terminal as more suitable than that was further away. This function appeared to have more of an impact than other tools on the output as the rail terminal in the County is in the northern part of the county while all of the land in the southern portion is less suitable.

After the railroad data was finished I then ran the slope tool on the DEM. I made it so the slopes that were lower were more suitable for mining while steeper slopes were not. After I ran the slope tool, I filtered output using the focal feature from the slope tool to make the map appear visibily more appealing. Once I finished, I performed a reclass on the water table feature class. The water table feature was not a raster so I used the topo-to-raster tool on the data. After the data was in raster format I performed a reclass which made areas closer to the water table more suitable than areas further away.

Figure 4: Suitability Map
 

Figure 5: Risk Map 

 

 In the second portion of our lab I was given the task of creating a map of impact/risk in Trempealeau County. This map can viewed in Figure 5 above. There were main pieces of data used in this investigation. They are as follows: proximity to streams, proximity to schools, proximity to wildlife areas, impact on residential areas, and impact on prime farmland. In order to get all of this data into raster format I used the Euclidean distance tool. This tool allowed me to get viable distances from these features that would consist of impact or lack of impact. I then had to reclassify all of the data into a form where they could be utilized to create a model. The final step to the risk evaluation came with the raster calculator. The final product can be seen in figure 5.

Figure 6: Viewshed of Weaver Hill in Trempealeau County

Figure 6 above introduces the next section of this lab where I was tasked with using an example of a prime recreational feature in our area of interest to run a viewshed on. As see above, from atop Weaver hill one can see a vast majority of the county. Through analysis of whether or not this region would be affected by suitable land to mine, it appeared that it would indeed. Some of the most suitable land for mining was actually located just around Weaver hill. With this said, if Weaver hill really is a prime recreational area then that means that some of the most suitable land for frac sand mining in the county could potentially be unavailable to private companies.

Figure 7: Trempealeau County Risk/Impact vs. Suitability Map

Figure 7 above represents both suitable areas of mining with proportional impact in the map as well. The final output shows areas in green where it is both suitable and has a relatively low risk. On the other hand, the regions in red show very unsuitable areas with high impact on the environment and people.

Conclusion

 This lab was the final installment of a series of investigation looking at frac sand mining in western Wisconsin. My study focused on Trempealeau County and helped me to further understand both the region and tools that help me to be a better geospatial analyst. This course endowed me with great skills in network analysis, raster/spatial analysis, etc. This project was very enjoyable to work on. Sand mining continues to be a rising, and often heated, topic in Wisconsin so learning more about it has given me a better view of both sides. In conclusion, Trempealeau County is a very diverse environment that contains a lot of great features, so hopefully the population stays informed as this topic emerges further.

Sources

http://dnr.wi.gov/maps/gis/datahydro.html

Network Analysis

Goals and Objectives

The goals of this lab were to use network analysis for the purpose of calculating the shortest distance from mines to railroad terminals and to get an estimate of the cost to ship the sand from a mine to a terminal.

The objectives were as follows:

1.       Load features into the Network Analysis Window

2.       Calculate a route

3.       Calculate a closest facility and route

4.       Build a model to calculate the closest facility route.

5.       Calculate the cost of sand truck travel on roads by county.

6.       Write a report (blog post) based on the results.

Methods

This lab required a lot of data from several sources. We utilized the mine data that we obtained from our previous exercise, railroad terminals (Prof. Hupy's folder) and the streets dataset from the ESRI resource. All of these main data, along with several other small figures, were used in a Network Analysis of the routes to take from each mine to a terminal. I used the closest facility function to receive the best route possible. The closest facility tool is composed of facilities and incidents which indicates a destination layer and source to work from. The last step is to have the tool solve the data. Next, the mileage of each route was calculated in order to create an index of how much it would cost to ship the sand 50 times back and forth at a cost of 2.2 cents per mile. Below is the equation used to determine this cost:

Costs = ([SUM_SHAPE_LENGTH]*0.022)*50


Many tools were used in the creation of this model. They consisted of:

     -Closest Facility
     -Add Locations
     -Solve
     -Copy Features
     -Intersect
     -Project
     -Summarize
     -Add Field
     -Calculate Field

Figure 1 above is the model that was utilized in the creation of the project. It was made in Model Builder.

Results

This lab used many tools to attempt to create a final product that was very relevant to our study. Through the usage of these tools the creation of an interesting spatial response was produced. Network Anlaysis can very important in find just not directions, but also paths of "least resistance." Figure 2 below shows the shortest path from mines to a designated rail terminal.

Figure 2 shows Frac Sand mines, railroad terminals, railroads and the route from a mine to a terminal. You can see a an outward spiral of mines from each terminal in the image. 

 



Figure 3 is shows the values that I received through my conversions and calculation of costs. The graph corresponds to the data in the table.



 

Conclusions

The lab was a great way to learn more about Network Analysis and many more very helpful tools. Some major tools that were showcased were Model Builder and Network Analysis tools (ie. closest facility and routes). My data would suggest that Trempealeau, Chippewa, La Crosse and Eau Claire counties all have the most traffic as far as Frac Sand is concerned. It will be interesting to compare data with my colleagues to see if we obtained similar results.

Data Downloading, Interoperability, and Working with Projections

Goal/Objectives

This lab was the one referenced in my first blog post that was, until recently, unaccessable due to the government shutdown. This was supposed to be the first part of a multi-leveled lab course aimed at the sustainability of Frac Sand mines in Tremealeau County, WI. This  lab is now part three in the installment. The purpose of this lab was to learn how to download data from online government sources, making these data collections interoperable and to fix them with popper projections. Several different government web sources were utilized in the creation of this lab. They were as follows: National Atlas, USGS National Map Viewer, Web Soil Survey, and the USDA.

General Methods

The general methods that were utilized here consisted of many different services. While downloading data from the online resources I had to use the web explorer device to unzip and save the data into my folder in our classes folder. Some of the sources allowed data transfer straight from their sites to my destination, while others offered an email containing the data. After all of the data was downloaded a personal geodatabase was created to store all of the tables and shapefiles. I assigned the NAD_1983(2011)_Wisconsin_TM in order to get the best picture of the whole state. I didn't only have information covering Trempealeau county so I had to use a common coordinate system for the whole state that would accurately portray all of the data. A feature dataset was also created to help in creating a relationship between the soils data and the component table. The soil data went under a process of joining both component data and a layer that contained a legend as well. Importing of a SSURGO geodatabase into access was also done to give the table a reference. Feature classes, tables and rasters were all imported into my ArcMap in order to create my final product. As all of the data was added to my database it was projected on the fly using my above coordinate system.

Results

The result of this project was several maps that were projected into the same coordinate system to best reflect the data in the Wisconsin region. The below data in Figure 2 shows the all of the data that I download: railroads, DEM, soils, NLCD, and NASS cropland. Figure 1 shows all of the data quality/metadata from the data that was downloaded as well.

Figure 1:

 

Figure 2:

Conclusion

So far in our investigation we have created the tools to have a strong database for analyzing the spatial connections of Wisconsin, it's mines, and the effects that they are having on people and the environment. We now have several databases full of data on the physical features of Wisconsin, along with knowledge of what frac sand mines are. It will be interesting to see how our investigation continues to progress as we compile and analyze data more acutely.

References
Lo, Chor Pang and Albert K.W. Yeung. "Concepts and Techniques of Geographic Information Systems." (2006).

Geocoding Frac Sand Mines in Wisconsin

Goal/Objective

This lab is an extension of the previous post and aims to show frac sand mining in Western Wisconsin through another lens. The goal of this lab was to download Trempealeau County data from the county website. The final result of this lab is a geocoded map created in ArcMap and an excel spreadsheet showing the distances of my points compared to other geocoded mines with the same address as mine.

1. Download Trempealeau County Land records data.
2. Download the updated list of mines from Wisconsin Watch.
3. Connect to the geocoding services provided by ESRI.
4. Geocode the downloaded mines to the street addresses that they area associated with.
5. Geocode the mines with the PLSS system manually.
6. Compare our results with our colleagues from class.

Methods

The initial part of this lab was to download Trempealeau County data from their geodatabase. The data was then utilized in our lab to concentrate on our region compared to the state. Trempealeau county has one of the highest concentrations of mines in Western Wisconsin and the data that was collected showed this. The next method used in this lab was to download data from our teacher's file which contained updated mines in Wisconsin. Each students in our course was assigned a unique number that corressponded with several mines. Then the mines were normalized in the excel file to make it plausible for them to be geocoded. The table was then put into ArcMap in a dbf file. Some of the addresses of the mines didn't match up correctly so you could either manually change the addresses or go with the suggested address of Arc. Once the geocoding was finished the class submitted their geocoded data in the form of a shapefile to the course folder. All of the data from the shapefiles were merged using ArcToolbox's merge tool. After geoprocessessing the mines were querried to find the mines with the same UNIQUE_ID as the mines that each individual originally geocoded. The querried mines were then created into their own feature. Finally, the point distance tool was used to find out how far our points were from our colleagues same mine points.

Results

In the figures below, you can see the excel files that were normalized, excel files with the finished product of distances and a map of the mines in Wisconsin. In Figure 1, the addresses were normalized to make an easier geocoding experience. The data given to us gave us some addresses or  hints to addresses but were not in a form that Arc could geocode. Figure 2, shows the distances and how geocoding and manually normalizing can create differences in points that have almost the same data. Finally, figure 3 is a map of the mines that I did and the ones that were the same as mine.

Figure 1:

Figure 2:

The distances above are in kilometers. The distances correspond to the space between my points and my colleagues.

Figure 3:

Discussion

Looking at Figure 3, it is very apparent of how manually normalizing data between individuals can create a distance divide in geocoded data. Because of the manual placing of addresses to mine sites when geocoding the ability to get perfectly correlated points between colleagues is nearly impossible. In the future I now have the ability to know more effective ways of normalizing data, especially when working with others. This lab also gave me the knowledge of how to work with others and finding a best way to normalize data where it will work with everyone's data. Both inherent and operational errors occur commonly according to Lo. Inherent errors are those in which occur during the creation of your data, while operational errors are those in which are created during work on your data. Because of these errors your data may not be fully correct and should be analyzed very closely. One example of an operational error is when I placed my mine inside of a certain county while my colleagues placed them elsewhere. Inherent errors would occur in my data if I didn't project my data in right coordinate system.

Conclusion

In our investigation of frac sand mines in Wisconsin we have so far learned about why frac sand is used, how to download data from government sites, and how to geocode addresses. Other skills that I have acquired from our labs include the ability to normalize data, how to navigate dbf files, how to convert dbf files into exel files, utilizing useful online source, comparing data to images, etc. As we continue discovering more about the sustainability of frac sand mines there will be more extensive data added to this blog.

Thank you for reading.

References:
Lo, Chor Pang and Albert K.W. Yeung. "Concepts and Techniques of Geographic Information Systems." 2 (2006): 108

Overview of Frac Sand Mining in Wisconsin

Overview of Frac Sand Mining in Wisconsin

Introduction

My Geographic Information Systems (GIS) 2 course will be examining frac sand mining in Wisconsin. We will be looking at Trempealeau county extensively throughout the semester. GIS will be used to model various components of the mining in Trempealeau country through mapping and reports. This post will cover an overview of what frac sand mining is, where it is in Wisconsin, and the issues associated with it.

What is frac sand mining?

Frac Sand mining has recently been a very heated topic in Wisconsin due to the state's reservoirs of fine grained quartz. While sand mining in Wisconsin has persisted for hundreds of years the industry has recently seen an explosion of mining proposals. Wisconsin sands are of high quality and fine grain, which in turn is the leading cause of the increase in permit requests. Once the sand is mined it is shipped out of state to be used by oil companies in Texas, North Dakota and Montana. The process that Oil fields use to pull the fuel from the earth's crust is called hydrofracking. Hydrofracking, hydraulic fracturing, is the process of drilling into the earth's surface, using explosives to break apart the rock and then surging water mixed with the frac sand which helps to hold the fractures open. The busting of the rock layers release natural gasses and liquids which are retrieved for commercial value.

Where is it in Wisconsin?

Western Wisconsin has the largest reservoirs of the silica sand that is needed for the hydrofracking process. Eau Claire, Chippewa, Trempealeau, and Monroe see a heavy amount of mining. Sand stone formations span across western and central Wisconsin where the concentration of mine sites are located. As stated at the beginning of this blog Trempealeau county is our studies focus and is also one of the best examples of a western Wisconsin county who is highly affects by sand mining. Trempealeau county, as seen in the figure below, has a very high concentration of both mines and processing plants. Sand from glacial and river beach deposits are generally too angular to be used in the hydrofracking process, so Wisconsin's sandstone mines are the best place to get pure sand. However not all sands are usable for hydrofracking. Natural resource companies look for a certain grain of sand to use.

Issues associated with Frac sand mining

In recent news the frac sand mines have been seen as more of an issue that in its several hundred year history. Regulators, local and state governments and the general public have all generated interest in this topic for their own reasons. The public's greatest concern appears to be the encroachment of mines into residential areas. Regulators want to create laws and guidelines for mining companies so as to limit the amount of sand taken. Sand mines fall under the jurisdiction of keeping water safe for drinking and also to keep hazardous materials out. Even though these laws are in place these issues still persist. The removal of overburden, which is the top of hills that covers the sandstone, has also met the public's eye because it is destroying the hills of western Wisconsin.

Using GIS to help observe some of these issues

Our course is designed for the enhancing of GIS knowledge, but our labs will also give us skills and knowledge of very important environmental issues in our region. We will also obtain the ability to download and use data from outside resources. We will be using ArcGIS to help in our investigation of Trempealeau counties frac sand mines. Lab 1 will be posted shortly, however because of the government shutdown my data download has been greatly hindered. I will post again as soon as possible!

Thank you for your attention!

http://dnr.wi.gov/topic/Mines/documents/SilicaSandMiningFinal.pdf
http://dnr.wi.gov/topic/mines/silica.html
http://dnr.wi.gov/topic/Mines/Deposit.html
http://wisconsingeologicalsurvey.org/pdfs/frac-sand-factsheet.pdf