00: Introduction to python for hydrologists¶
In this background introduction, we will review concepts introduced in the University of Waterloo Python Background Tutorial
This notebook is a general overview of Python. We will cover many of the basic aspects of Python that are distinct (and sometimes confusing) relative to other programming and scripting languages.
First let’s consider three ways that we can interact with Python:
Through a command-line session
Using iPython
Using a Jupyter notebook like this one
[1]:
print('hello world')
hello world
Using Python as a Calculator¶
Python can be used to simple perform arithmetic operations. (note: in an iPython notebook like this one, shift-ENTER evaluates the code)
Spaces are fine
[2]:
6*9
[2]:
54
[3]:
6 + (99.98 * 5.)
[3]:
505.90000000000003
Exponents use ** rather than ^ in some other languages.
[4]:
print (5**9.1)
2294177.623218786
[ ]:
What if you want to do more than basic arithmetic?¶
There are a couple ways to access more mathematical functions:
[5]:
import numpy as np
import math
np.sqrt(9)
[5]:
np.float64(3.0)
[6]:
? np
[7]:
math.sqrt(9.1)
[7]:
3.0166206257996713
Of these two options, we typically use np because, as we will cover in detail later, numpy provides many other features important to the kind of scientific work we perform.
Data Structures and a few definitions¶
mutability – An immutable object is one whose property or state cannot be changed, whereas, mutable objects can change state
integer (int)float/double precision (float)long ingteger (long)complex (complex) There are also groups of characters: string (str)Variables can be of any type and the results can be assigned to symbols without predefining type (like DIMENSION in FORTRAN). print statements display values to standard output, and the type function tells you which number type a variable (or number) is.
[8]:
a = 5.9
print (a)
aa = 5.91
print (a)
print (aa)
5.9
5.9
5.91
[9]:
a2 = 5.
print (a2)
type(a2)
5.0
[9]:
float
Variables and types¶
Symbol names¶
Variable names in Python can contain alphanumerical characters a-z, A-Z, 0-9 and some special characters such as _. Normal variable names must start with a letter.
By convention, variable names start with a lower-case letter, and Class names start with a capital letter.
In addition, there are a number of Python keywords that cannot be used as variable names. These keywords are:
and, as, assert, break, class, continue, def, del, elif, else, except, exec, finally, for, from, global, if, import, in, is, lambda, not, or, pass, print, raise, return, try, while, with, yield
Note: Be aware of the keyword lambda, which could easily be a natural variable name in a scientific program. But being a keyword, it cannot be used as a variable name.
[10]:
import keyword
keyword.kwlist
[10]:
['False',
'None',
'True',
'and',
'as',
'assert',
'async',
'await',
'break',
'class',
'continue',
'def',
'del',
'elif',
'else',
'except',
'finally',
'for',
'from',
'global',
'if',
'import',
'in',
'is',
'lambda',
'nonlocal',
'not',
'or',
'pass',
'raise',
'return',
'try',
'while',
'with',
'yield']
There are also built-in objects that should not be used as variable names.
[11]:
import builtins
dir(builtins)
[11]:
['ArithmeticError',
'AssertionError',
'AttributeError',
'BaseException',
'BaseExceptionGroup',
'BlockingIOError',
'BrokenPipeError',
'BufferError',
'BytesWarning',
'ChildProcessError',
'ConnectionAbortedError',
'ConnectionError',
'ConnectionRefusedError',
'ConnectionResetError',
'DeprecationWarning',
'EOFError',
'Ellipsis',
'EncodingWarning',
'EnvironmentError',
'Exception',
'ExceptionGroup',
'False',
'FileExistsError',
'FileNotFoundError',
'FloatingPointError',
'FutureWarning',
'GeneratorExit',
'IOError',
'ImportError',
'ImportWarning',
'IndentationError',
'IndexError',
'InterruptedError',
'IsADirectoryError',
'KeyError',
'KeyboardInterrupt',
'LookupError',
'MemoryError',
'ModuleNotFoundError',
'NameError',
'None',
'NotADirectoryError',
'NotImplemented',
'NotImplementedError',
'OSError',
'OverflowError',
'PendingDeprecationWarning',
'PermissionError',
'ProcessLookupError',
'RecursionError',
'ReferenceError',
'ResourceWarning',
'RuntimeError',
'RuntimeWarning',
'StopAsyncIteration',
'StopIteration',
'SyntaxError',
'SyntaxWarning',
'SystemError',
'SystemExit',
'TabError',
'TimeoutError',
'True',
'TypeError',
'UnboundLocalError',
'UnicodeDecodeError',
'UnicodeEncodeError',
'UnicodeError',
'UnicodeTranslateError',
'UnicodeWarning',
'UserWarning',
'ValueError',
'Warning',
'ZeroDivisionError',
'__IPYTHON__',
'__build_class__',
'__debug__',
'__doc__',
'__import__',
'__loader__',
'__name__',
'__package__',
'__spec__',
'abs',
'aiter',
'all',
'anext',
'any',
'ascii',
'bin',
'bool',
'breakpoint',
'bytearray',
'bytes',
'callable',
'chr',
'classmethod',
'compile',
'complex',
'copyright',
'credits',
'delattr',
'dict',
'dir',
'display',
'divmod',
'enumerate',
'eval',
'exec',
'execfile',
'filter',
'float',
'format',
'frozenset',
'get_ipython',
'getattr',
'globals',
'hasattr',
'hash',
'help',
'hex',
'id',
'input',
'int',
'isinstance',
'issubclass',
'iter',
'len',
'license',
'list',
'locals',
'map',
'max',
'memoryview',
'min',
'next',
'object',
'oct',
'open',
'ord',
'pow',
'print',
'property',
'range',
'repr',
'reversed',
'round',
'runfile',
'set',
'setattr',
'slice',
'sorted',
'staticmethod',
'str',
'sum',
'super',
'tuple',
'type',
'vars',
'zip']
Assignment¶
The assignment operator in Python is =. Python is a dynamically typed language, so we do not need to specify the type of a variable when we create one.
Assigning a value to a new variable creates the variable:
[12]:
b=99.1
# variable names can practically as long as you like
this_reallyDESCRIPTIVE_crazyvariableNAME_is_maybe_a_bit_long = 7
print(this_reallyDESCRIPTIVE_crazyvariableNAME_is_maybe_a_bit_long)
7
In general, appropriate type conversion is possible:
[13]:
a = 8.9
b = int(a)
print (b)
c = str(a)
b
8
[13]:
8
Strings¶
Grouping characters together creates strings.
[14]:
avar = "this is a string (Includes 'spaces' and punctuation)"
print(avar)
len(avar)
this is a string (Includes 'spaces' and punctuation)
[14]:
52
Using square brackets ([]) we can dereference values as if the string were an array or a list (as we will discuss next)
[15]:
print(avar[0])
print(avar[1])
# negative indices count from the end of the string
print(avar[-1])
print(avar[-2])
# we can even print every nth character
print(avar[::3])
t
h
)
n
tss rgIle'asa nui)
Since strings are just sequences of characters, they can be operated on using addition and multiplication.
[16]:
d = 'a' + 'b' + ' c '
print (d)
print (d*5)
ab c
ab c ab c ab c ab c ab c
For comparisons, it can be important to match case, so we can cast to upper or lower or a couple other options
[17]:
avar.upper()
[17]:
"THIS IS A STRING (INCLUDES 'SPACES' AND PUNCTUATION)"
[18]:
avar.lower()
[18]:
"this is a string (includes 'spaces' and punctuation)"
[19]:
avar.capitalize()
[19]:
"This is a string (includes 'spaces' and punctuation)"
[20]:
avar.swapcase()
[20]:
"THIS IS A STRING (iNCLUDES 'SPACES' AND PUNCTUATION)"
We can also replace text using replace()
[21]:
avar.replace('and','&')
[21]:
"this is a string (Includes 'spaces' & punctuation)"
Lists¶
There are a few ways to group values in Python. The most basic way is a list, defined by square brackets([ ]) and comma-delimited
[22]:
a = [2.3, 3.4, 5.5, 12.4, 33]
print(type(a))
print(type(a[1]))
print(a)
<class 'list'>
<class 'float'>
[2.3, 3.4, 5.5, 12.4, 33]
Important Note!!! –>¶
Python uses zero-referencing (like C, but unlike MATLAB and FORTRAN). Using square brackets, look at the various values of a (e.g. a[0], a[2]). Also, see what you get with a[-1] and a[-2]
[23]:
a[0]
[23]:
2.3
Lists of Lists¶
You can make lists made up of anything (e.g. numbers, other lists, etc.)
[24]:
b=5
c=[3,'4stuff',5.9]
d = [a,b,c]
print (d)
print (d[0])
print (d[1])
print (d[2])
[[2.3, 3.4, 5.5, 12.4, 33], 5, [3, '4stuff', 5.9]]
[2.3, 3.4, 5.5, 12.4, 33]
5
[3, '4stuff', 5.9]
[25]:
aaaa = [1,2,3]
aaaa + [5]
[25]:
[1, 2, 3, 5]
append versus extend¶
You can add to a list in two ways – using the extend or append methods. Let’s look at append first:
[26]:
a=[1,2,3]
print (a)
[1, 2, 3]
[27]:
a.append(4)
a.append(5)
print (a)
[1, 2, 3, 4, 5]
[28]:
a.append([6,7,8])
print (a)
a[-1]
[1, 2, 3, 4, 5, [6, 7, 8]]
[28]:
[6, 7, 8]
This did exactly what it should, but looks strange at first. We appended on the data we wanted to, but since we passed a list, we just took the list [6,7,8] on to the end of a. What if we wanted to continue our sequence? Then we could use extend.
[29]:
a=[1,2,3,4,5]
a.extend([6,7,8])
print (a)
[1, 2, 3, 4, 5, 6, 7, 8]
The same behavior will happen even if the list as it is is already made up of multiple types.
[30]:
print (d)
d.append([5,6,7,8,9])
print (d)
[[2.3, 3.4, 5.5, 12.4, 33], 5, [3, '4stuff', 5.9]]
[[2.3, 3.4, 5.5, 12.4, 33], 5, [3, '4stuff', 5.9], [5, 6, 7, 8, 9]]
We can create an empty list two ways
[31]:
a = list()
print (type(a), a)
b = []
print (type(b), b)
<class 'list'> []
<class 'list'> []
[32]:
a.append('stuff')
a.append(99)
print (a)
['stuff', 99]
[33]:
aaa = [0]
aaa.append(8)
aaa
[33]:
[0, 8]
We can also add and remove elements from lists in a couple ways:
[34]:
a
[34]:
['stuff', 99]
[35]:
a = [ 2,99,2,99, 'stuff']
print (a.index(99))
a
1
[35]:
[2, 99, 2, 99, 'stuff']
[36]:
# index tells us where in a list a specific value resides
print (a.index(99))
print (a.index('stuff'))
# insert can add a value -- note that this shifts the list over
a.insert(0, 'newstuff')
print (a)
1
4
['newstuff', 2, 99, 2, 99, 'stuff']
[37]:
a[0] = 99999
print (a)
b = a.pop(4) #remove item in position 4 and assign to b
print (b)
a
[99999, 2, 99, 2, 99, 'stuff']
99
[37]:
[99999, 2, 99, 2, 'stuff']
Tuples¶
A similar construction to a list is a tuple. A tuple is defined using regular parentheses (()). The key differences is tuples are immutable.
[38]:
# we can make a list
a = [3,4,5]
print (type(a))
print (a)
# it's mutable meaning we can change it in place
a[0] = 999
print (a)
<class 'list'>
[3, 4, 5]
[999, 4, 5]
[39]:
# a tuple is similar
a = (3,4,5)
print(type(a))
print (a)
# we can dereference values from it in the same way as we would a list
print (a[0])
# but we cannot change values in place
a[0] = 999
<class 'tuple'>
(3, 4, 5)
3
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[39], line 8
6 print (a[0])
7 # but we cannot change values in place
----> 8 a[0] = 999
TypeError: 'tuple' object does not support item assignment
[40]:
a=(1,'howdy')
b=list(a)
print (type(a))
print (type(b))
<class 'tuple'>
<class 'list'>
We won’t talk very much about tuples except to say that they are sometimes required as input to functions, sometimes functions return them, and they can be useful for other reasons. Also, they can be constructed using zip to pull together a couple lists.
[41]:
a = zip([1,2],[11,22],[111,222])
print (a)
print(type(a))
for b in a:
print(b)
<zip object at 0x7f09b4a71880>
<class 'zip'>
(1, 11, 111)
(2, 22, 222)
We can also assign tuples without using parentheses, although it’s kind of sloppy. We can unpack a tuple into multiple variables.
[42]:
a = 5,6,7
val1, val2, val3 = a
print (a)
print (val1)
print (val2)
print (val3)
(5, 6, 7)
5
6
7
This can be done all at once as well!
[43]:
val1, val2, val3 = 5,6,7
print (val1)
print (val2)
print (val3)
5
6
7
Dictionaries¶
Dictionaries are also like lists, except that each element is a key-value pair. The syntax for dictionaries is {key1 : value1, ...}:
[44]:
params = {"parameter1" : 1.0,
"parameter2" : 2.0,
"parameter3" : 3.0,}
print(type(params))
print(params)
parlist = [ 1.0, 2.0, 3.0]
print (params)
<class 'dict'>
{'parameter1': 1.0, 'parameter2': 2.0, 'parameter3': 3.0}
{'parameter1': 1.0, 'parameter2': 2.0, 'parameter3': 3.0}
[45]:
params.items()
[45]:
dict_items([('parameter1', 1.0), ('parameter2', 2.0), ('parameter3', 3.0)])
The keys can be obtained separate from their values as a list.
[46]:
for i in params.keys():
print (i)
parameter1
parameter2
parameter3
[47]:
ckeys = [i for i in params.keys()]
[48]:
params[ckeys[1]]
[48]:
2.0
[49]:
print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))
parameter1 = 1.0
parameter2 = 2.0
parameter3 = 3.0
Can also get just the values
[50]:
params.values()
[50]:
dict_values([1.0, 2.0, 3.0])
Or everyone’s favorite….tuples! .items() returns tuples of the key, values pairs
[51]:
params.items()
[51]:
dict_items([('parameter1', 1.0), ('parameter2', 2.0), ('parameter3', 3.0)])
[52]:
params["parameter1"] = "A"
params["parameter2"] = "B"
# add a new entry
params["parameter4"] = "D"
print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))
print("parameter4 = " + str(params["parameter4"]))
parameter1 = A
parameter2 = B
parameter3 = 3.0
parameter4 = D
A trick using zip and dict¶
Say we have two lists – one of which is a set of keys and another is accompanying values (like we read them in from two columns in a spreadsheet)
[53]:
keys = ['HK','VKA','RCH','COND']
vals = [1.2e-6, 0.001, 1.9e-9, 4.0]
MFdict = dict(zip(keys,vals))
MFdict
[53]:
{'HK': 1.2e-06, 'VKA': 0.001, 'RCH': 1.9e-09, 'COND': 4.0}
Process Flow¶
We will cover conditional statements and a variety of loop types. ### Conditional statements: if, elif, else
The Python syntax for conditional execution of code use the keywords if, elif (else if), else:
[54]:
statement1 = True
statement2 = True
if statement1==True:
print("statement1 is True")
elif statement2==True:
print("statement2 is True")
else:
print("statement1 and statement2 are False")
statement1 is True
There are a couple critical things to see here.
There is no
end ifor other closing of the statement. This is all controlled by indentation.Each statement ends in a
':'
[55]:
statement1 or statement2
[55]:
True
[56]:
not statement2
[56]:
False
Rather than just True or False, we can check for equivalence to other values.
‘
==’ indicates test for equality‘
!=’ indicates test for nonequality
[57]:
a = 5
print (a)
a!=5.01
5
[57]:
True
[58]:
statement1 = 5.2
if statement1 == 'junk':
print ('junky')
elif statement1 != 5:
print ('neither "junky" nor 5')
else:
print ('must be 5 eh?')
print ('wow - it worked?')
neither "junky" nor 5
wow - it worked?
An aside on indentation¶
Technically, tabs or any consistent number of spaces can be used to control indentation. A casual standard is using 4 spaces. Many editors replace tabs with spaces to be safe.
The level of indentation defines blocks without requiring a symbol or word to close a block or loop.
few examples of indentation behavior:¶
[59]:
statement1 = True
statement2 = True
if statement1==True:
if statement2==True:
print("both statement1 and statement2 are True")
print( statement2)
both statement1 and statement2 are True
True
[60]:
# Bad indentation!
if statement1==True:
if statement2==True:
print("both statement1 and statement2 are True") # this line is not properly indented
Cell In[60], line 4
print("both statement1 and statement2 are True") # this line is not properly indented
^
IndentationError: expected an indented block after 'if' statement on line 3
[61]:
statement1 = True
if statement1==True:
print("printed if statement1 is True")
print("Trying to say False but still inside the if block")
printed if statement1 is True
Trying to say False but still inside the if block
[62]:
if statement1:
print("printed if statement1 is True")
print("Statement 1 is false --- printed now outside the if block")
printed if statement1 is True
Statement 1 is false --- printed now outside the if block
Loops¶
In Python, loops can be programmed in a number of different ways. The most common is the for loop, which is used together with iterable objects, such as lists. The basic syntax is:
``for`` loops:
[63]:
for i in [1,2,[9,8,0],4]:
print (i)
1
2
[9, 8, 0]
4
The for loop iterates over the elements of the supplied list, and executes the containing block once for each element. Note the same indentation rules as for if statements.
Any kind of list can be used in the for loop. For example:
[64]:
range(4)
[64]:
range(0, 4)
[65]:
for x in range(4): # by default range start at 0
print(x)
print ('-'*12)
print ('4 is missing but the length is {0}'.format(len(range(4))))
0
1
2
3
------------
4 is missing but the length is 4
Uh oh! Where is 4? range constructions do not include the final value. This takes some getting used to!
There is a long discussion here: http://stackoverflow.com/questions/4504662/why-does-rangestart-end-not-include-end
Suffice it to say that the length of a range is the number stated and this is all related to zero indexing.
Ranges can also start at values other than 0, but still never include the final value.
[66]:
for x in range(-3,3):
print(x)
-3
-2
-1
0
1
2
Notice that we aren’t using an index like i to dereference items from the list. Rather, we are just iterating over those items. They need not be numbers.
[67]:
a = ['This', 'is', 'different', 'than', 'F77']
for word in a:
print (word)
This
is
different
than
F77
We could do this like in other languages, but quickly it becomes clear that it would be contrived.
[68]:
for i in range(len(a)):
print (a[i])
This
is
different
than
F77
Sometimes we also need an index for some reason. We can get both using enumerate
[69]:
for i, word in enumerate(a):
print ('word number {0} is "{1}"'.format(i, word))
word number 0 is "This"
word number 1 is "is"
word number 2 is "different"
word number 3 is "than"
word number 4 is "F77"
We can also iterate over the keys of a dictionary in a couple ways. First, just iterating over the keys:
[70]:
for key in params.keys():
print (key + " = " + str(params[key]))
parameter1 = A
parameter2 = B
parameter3 = 3.0
parameter4 = D
[71]:
# the following shorthand is now preferred
for key in params:
print (key + " = " + str(params[key]))
parameter1 = A
parameter2 = B
parameter3 = 3.0
parameter4 = D
Second we can iterate over the items and unpack them.
[72]:
for key, val in params.items():
print (key + " = " + str(val))
#print (poo)
parameter1 = A
parameter2 = B
parameter3 = 3.0
parameter4 = D
List Comprehension¶
A very Pythonic and compact way to build a list using a for loop is using a list comprehension
[73]:
a = [x**3 for x in range(10)]
print (a)
[0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
You can get conditional statements into a list comprehension – but, once it get too complicated, it gets to be “clever” and it’s better to be more explicit
[74]:
b = [x**3 if x>0 else x for x in range(-5,5)]
print (b)
[-5, -4, -3, -2, -1, 0, 1, 8, 27, 64]
[75]:
adict = dict(zip(range(10),a))
[76]:
for key, value in adict.items():
print (str(key) + "^3 = " + str(value))
0^3 = 0
1^3 = 1
2^3 = 8
3^3 = 27
4^3 = 64
5^3 = 125
6^3 = 216
7^3 = 343
8^3 = 512
9^3 = 729
[77]:
d = {str(a) + "^3": a**3 for a in range(10)}
print (d)
{'0^3': 0, '1^3': 1, '2^3': 8, '3^3': 27, '4^3': 64, '5^3': 125, '6^3': 216, '7^3': 343, '8^3': 512, '9^3': 729}
``while`` loops:
[78]:
i = 0
while i<4:
print(i)
i += 1 # shorthand for i = i + 1
0
1
2
3
[79]:
i = 5
while i: # shorthand for while i != 0
print('howdy again')
i -= 1 # shorthand for i = i - 1
howdy again
howdy again
howdy again
howdy again
howdy again
[80]:
i = 0
while 3:
print(i)
i += 2
if i > 10:
break
0
2
4
6
8
10
HEEEEEELP!¶
You will always need help with Python. There are so many packages, modules, functions, datatypes, and everything else it takes a while to keep it all straight, and you will always need to update
Stackoverflow is amazing–it can be toxic at times, but the community often converges on useful answers. Here’s an example.
Official documentation is good, but can overwhelm.
Python in general: https://docs.python.org/3/contents.html
numpy and scipy: http://docs.scipy.org/doc/
matplotlib: https://matplotlib.org/stable/users/index.html and especially https://matplotlib.org/stable/gallery/index.html
Of course, Google is good Google your error message text!
If you like books, consider O’Reilly
While in iPython (or a notebook) try
help(os)or? osfor example.help(os)brings up docstrings in the output window?osopens a separate pane below (that you can kick out to another tab using a button next to the X)
what about chatGPT? For sure, you can try to get chatGPT to write your code. It’s pretty effective, but definitely you need to verify what it suggests!
[ ]: