Database
A Database is a collection of datasets and other CDMS objects. It consists of a hierarchical collection of objects, with the database being at the root, or top of the hierarchy. A database is used to:
-
search for metadata
-
access data
-
provide authentication and access control for data and metadata
The figure below illustrates several important points:
-
Each object in the database has a relative name of the form tag=id. The id of an object is unique with respect to all objects contained in the parent.
-
The name of the object consists of its relative name followed by the relative name(s) of its antecedent objects, up to and including the database name. In the figure below, one of the variables has name
"variable=ua, dataset=ncep_reanalysis_mo,database=CDMS".
-
Subordinate objects are thought of as being contained in the parent. In this example, the database `CDMS' contains two datasets, each of which contain several variables.
Overview
To access a database:
-
Open a connection. The connect method opens a database connection. connect takes a database URI and returns a database object:
db = cdms.connect("ldap://dbhost.llnl.gov/database=CDMS,ou=PCMDI,o=LLNL,c=US")
-
Search the database, locating one or more datasets, variables, and/or other objects.
The database searchFilter method searches the database. A single database connection may be used for an arbitrary number of searches.
For example, to find all observed datasets:
result = db.searchFilter("category=observed",tag="dataset")
Searches can be restricted to a subhierarchy of the database. This example searches just the dataset `ncep_reanalysis_mo':
result = db.searchFilter(relbase="dataset=ncep_reanalysis")
-
Refine the search results if necessary. The result of a search can be narrowed with the searchPredicate method.
-
Process the results. A search result consists of a sequence of entries. Each entry has a name, the name of the CDMS object, and an attribute dictionary, consisting of the attributes located by the search:
for entry in result:
print entry.name, entry.attributes
-
Access the data. The CDMS object associated with an entry is obtained from the getObject method:
obj = entry.getObject()
If the id of a dataset is known, the dataset can be opened directly with the open method:
dset = db.open("ncep_reanalysis_mo")
-
Close the database connection:
db.close()
Database Internal Attributes
|
Type
|
Name
|
Summary
|
|
Dictionary
|
attributes
|
Database attribute dictionary
|
|
LDAP
|
db
|
(LDAP only) LDAP database object
|
|
String
|
netloc
|
Hostname, for server-based databases
|
|
String
|
path
|
path name
|
|
String
|
uri
|
Uniform Resource Identifier.
|
Database Constructors
|
|
|
db = cdms.connect(uri=None, user="", password="")
Connect to the database.
uri is the Universal Resource Indentifier of the database. The form of the URI depends on the implementation of the database. For a Lightweight Directory Access Protocol (LDAP) database, the form is:
ldap://host[:port]/dbname
For example, if the database is located on host `dbhost.llnl.gov', and is named 'database=CDMS,ou=PCMDI,o=LLNL,c=US', the URI is:
ldap://dbhost.llnl.gov/database=CDMS,ou=PCMDI,o=LLNL,c=US
If unspecified, the URI defaults to the value of environment variable CDMSROOT.
user is the user ID. If unspecified, an anonymous connection is made.
password is the user password. A password is not required for an anonymous connection.
|
Database Methods
|
Type
|
Definition
|
|
None
|
close()
Close a database connection.
|
|
List
|
listDatasets()
Return a list of the dataset IDs in this database. A dataset ID can be passed to the
open
command.
|
|
Dataset
|
open(dsetid, mode='r')
Open a dataset.
dsetid is the string dataset identifier
mode is the open mode, 'r' - read-only, 'r+' - read-write, 'w' - create.
openDataset
is a synonym for
open
.
|
|
SearchResult
|
searchFilter(filter=None, tag=None, relbase=None, scope=Subtree, attnames=None, timeout=None)
Search a CDMS database.
filter is the string search filter. Simple filters have the form "tag = value". Simple filters can be combined using logical operators '&', '|', '!' in prefix notation. For example, the filter '(&(objectclass=variable)(id=cli))' finds all variables named "cli". A formal definition of search filters is provided in the following section.
tag restricts the search to objects with that tag ("dataset" | "variable" | "database" | "axis" | "grid").
relbase is the relative name of the base object of the search. The search is restricted to the base object and all objects below it in the hierarchy. For example, to search only dataset `ncep_reanalysis_mo', specify:
relbase="dataset=ncep_reanalysis_mo".
To search only variable 'ua' in ncep_reanalysis_mo, use:
relbase="variable=ua, dataset=ncep_reanalysis_mo"
If no base is specified, the entire database is searched. See the scope argument also.
scope is the search scope (Subtree | Onelevel | Base). Subtree searches the base object and its descendants. Onelevel searches the base object and its immediate descendants. Base searches the base object alone. The default is Subtree.
attnames: list of attribute names. Restricts the attributes returned. If None, all attributes are returned. Attributes 'id' and 'objectclass' are always included in the list.
timeout: integer number of seconds before timeout. The default is no timeout.
|
Searching a database
The searchFilter method is used to search a database. The result is called a search result, and consists of a sequence of result entries.
In its simplest form, searchFilter takes an argument consisting of a string filter. The search returns a sequence of entries, corresponding to those objects having an attribute which matches the filter. Simple filters have the form (tag = value), where value can contain wildcards. For example:
'(id = ncep*)'
'(project = AMIP2)'
Simple filters can be combined with the logical operators `&', `|', `!'. For example,
'(&(id = bmrc*)(project = AMIP2))'
matches all objects with id starting with 'bmrc', and a 'project' attribute with value 'AMIP2'.
Formally, search filters are strings defined as follows:
filter ::= "(" filtercomp ")"
filtercomp ::= "&" filterlist | # and
"|" filterlist | # or
"!" filterlist | # not
simple
filterlist ::= filter | filter filterlist
simple ::= tag op value
op ::= "=" | # equality
"~=" | # approximate equality
"<=" | # lexicographically less than or equal to
">=" # lexicographically greater than or equal to
tag ::= string attribute name
value ::= string attribute value, may include '*' as a wild card
Attribute names are defined in the chapter on Climate Data Markup Language (CDML). In addition, some special attributes are defined for convenience:
-
category is either "experimental" or "observed"
-
parentid is the ID of the parent dataset
-
project is a project identifier, e.g., "AMIP2"
-
objectclass is the list of tags associated with the object.
The set of objects searched is called the search scope. The top object in the hierarchy is the base object. By default, all objects in the database are searched, that is, the database is the base object. If the database is very large, this may result in an unnecessarily slow or inefficient search. To remedy this the search scope can be limited in several ways:
-
The base object can be changed.
-
The scope can be limited to the base object and one level below, or to just the base object.
-
The search can be restricted to objects of a given class (dataset, variable, etc.)
-
The search can be restricted to return only a subset of the object attributes
-
The search can be restricted to the result of a previous search.
A search result is accessed sequentially within a for loop:
result = db.searchFilter('(&(category=obs*)(id=ncep*))')
for entry in result:
print entry.name
Search results can be narrowed using searchPredicate. In the following example, the result of one search is itself searched for all variables defined on a 94x192 grid:
>>> result = db.searchFilter('parentid=ncep*',tag="variable")
>>> len(result)
65
>>> result2 = result.searchPredicate(lambda x: x.getGrid().shape==(94,192))
>>> len(result2)
3
>>> for entry in result2: print entry.name
variable=rluscs,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
variable=rlds,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
variable=rlus,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
>>>
SearchResult Methods
|
Type
|
Definition
|
|
ResultEntry
|
[i]
Return the i-th search result. Results can also be returned in a for loop:
for entry in db.searchResult(tag="dataset"):
...
|
|
Integer
|
len()
Number of entries in the result.
|
|
SearchResult
|
searchPredicate(predicate, tag=None)
Refine a search result, with a predicate search.
predicate is a function which takes a single CDMS object and returns true (1) if the object satisfies the predicate, 0 if not.
tag restricts the search to objects of the class denoted by the tag.
Note: In the current implementation, searchPredicate is much less efficient than searchFilter. For best performance, use searchFilter to narrow the scope of the search, then use searchPredicate for more general searches.
|
A search result is a sequence of result entries. Each entry has a string name, the name of the object in the database hierarchy, and an attribute dictionary. An entry corresponds to an object found by the search, but differs from the object, in that only the attributes requested are associated with the entry. In general, there will be much more information defined for the associated CDMS object, which is retrieved with the getObject method.
ResultEntry Attributes
|
Type
|
Name
|
Summary
|
|
String
|
name
|
The name of this entry in the database.
|
|
Dictionary
|
attributes
|
The attributes returned from the search.
attributes[key] is a list of all string values associated with the key.
|
ResultEntry Methods
|
Type
|
Definition
|
|
CdmsObj
|
getObject()
Return the CDMS object associated with this entry.
Note: For many search applications it is unnecessary to access the associated CDMS object. For best performance this function should be used only when necessary, for example, to retrieve data associated with a variable.
|
Accessing data
To access data via CDMS:
-
Locate the dataset ID. This may involve searching the metadata.
-
Open the dataset, using the open method.
-
Reference the portion of the variable to be read.
In the next example, a portion of variable 'ua' is read from dataset 'ncep_reanalysis_mo':
dset = db.open('ncep_reanalysis_mo')
ua = dset.variables['ua']
data = ua[0,0]
Examples of database searches
In the following examples, db is the database opened with
db = cdms.connect()
This defaults to the database defined in environment variable CDMSROOT.
List all variables in dataset 'ncep_reanalysis_mo':
for entry in db.searchFilter(filter="parentid=ncep_reanalysis_mo", tag="variable"):
print entry.name
Find all axes with bounds defined:
for entry in db.searchFilter(filter="bounds=*",tag="axis"):
print entry.name
Locate all GDT datasets:
for entry in db.searchFilter(filter="Conventions=GDT*",tag="dataset"):
print entry.name
Find all variables with missing time values, in observed datasets:
def missingTime(obj):
time = obj.getTime()
return time.length != time.partition_length
result = db.searchFilter(filter="category=observed")
for entry in result.searchPredicate(missingTime):
print entry.name
Find all CMIP2 datasets having a variable with id "hfss":
for entry in db.searchFilter(filter="(&(project=CMIP2)(id=hfss))",tag="variable"):
print entry.getObject().parent.id
Find all observed variables on 73x144 grids:
result = db.searchFilter('category=obs*')
for entry in result.searchPredicate(lambda x: x.getGrid().shape==(73,144),tag="variable"):
print entry.name
Find all observed variables with more than 1000 timepoints:
result = db.searchFilter('category=obs*')
for entry in result.searchPredicate(lambda x: len(x.getTime())>1000, tag="variable"):
print entry.name, len(entry.getObject().getTime())
Find the total number of each type of object in the database
print len(db.searchFilter(tag="database")),"database"
print len(db.searchFilter(tag="dataset")),"datasets"
print len(db.searchFilter(tag="variable")),"variables"
print len(db.searchFilter(tag="axis")),"axes"
MV module
The fundamental CDMS data object is the variable. A variable is comprised of:
-
a masked data array, as defined in the NumPy MA module.
-
a domain: an ordered list of axes and/or grids.
-
an attribute dictionary.
The MV module is a work-alike replacement for the MA module, that carries along the domain and attribute information where appropriate. MV provides the same set of functions as MA. However, MV functions generate transient variables as results. Often this simplifies scripts that perform computation. MA is part of the Python Numeric package, documented at http://numpy.sourceforge.net.
MV can be imported with the command:
import MV
The command
from MV import *
allows use of MV commands without any prefix.
Table 2.25 lists the constructors in MV. All functions return a transient variable. In most cases the keywords axes, attributes, and id are available. axes is a list of axis objects which specifies the domain of the variable. attributes is a dictionary. id is a special attribute string that serves as the identifier of the variable, and should not contain blanks or non-printing characters. It is used when the variable is plotted or written to a file. Since the id is just an attribute, it can also be set like any attribute:
var.id = 'temperature'
For completeness MV provides access to all the MA functions. The functions not listed in the following tables are identical to the corresponding MA function: allclose, allequal, common_fill_value, compress, create_mask, dot, e, fill_value, filled, get_print_limit, getmask, getmaskarray, identity, indices, innerproduct, isMA, isMaskedArray, is_mask, isarray, make_mask, make_mask_none, mask_or, masked, pi, put, putmask, rank, ravel, set_fill_value, set_print_limit, shape, size, sort. See the documentation at http://numpy.sourceforge.net for a description of these functions.
Variable Constructors in module MV
|
|
|
arrayrange(start, stop=None, step=1, typecode=None, axis=None, attributes=None, id=None)
Just like MA.arange() except it returns a variable whose type can be specfied by the keyword argument typecode. The axis, attribute dictionary, and string identifier of the result variable may be specified.
Synonym: arange
|
|
masked_array(a, mask=None, fill_value=None, axes=None, attributes=None, id=None)
Same as MA.masked_array but creates a variable instead. If no axes are specified, the result has default axes, otherwise axes is a list of axis objects matching a.shape.
|
|
masked_object(data, value, copy=1, savespace=0, axes=None, attributes=None, id=None)
Create variable masked where exactly data equal to value. Create the variable with the given list of axis objects, attribute dictionary, and string id.
|
|
masked_values(data, value, rtol=1e-05, atol=1e-08, copy=1, savespace=0, axes=None, attributes=None, id=None)
Constructs a variable with the given list of axes and attribute dictionary, whose mask is set at those places where
abs (data - value) < atol + rtol * abs (data).
This is a careful way of saying that those elements of the data that have value = value (to within a tolerance) are to be treated as invalid. If data is not of a floating point type, calls masked_object instead.
|
|
ones(shape, typecode='l', savespace=0, axes=None, attributes=None, id=None)
Return an array of all ones of the given length or shape.
|
|
reshape(a, newshape, axes=None, attributes=None, id=None)
Copy of a with a new shape.
|
|
resize(a, new_shape, axes=None, attributes=None, id=None)
Return a new array with the specified shape. The original array's total size can be any size.
|
|
zeros(shape, typecode='l', savespace=0, axes=None, attributes=None, id=None)
An array of all zeros of the given length or shape.
|
The following table describes the MV non-constructor functions. With the exception of argsort, all functions return a transient variable.
MV functions
|
Definition
|
|
argsort(x, axis=-1, fill_value=None)
Return a Numeric array of indices for sorting along a given axis.
|
|
asarray(data, typecode=None)
Same as cdms.createVariabledata, typecode, copy=0). This is a short way of ensuring that something is an instance of a variable of a given type before proceeding, as in
data = asarray(data)
Also see the variable astype() function.
|
|
average(a, axis=0, weights=None)
computes the average value of the non-masked elements of x along the se-lected axis. If weights is given, it must match the size and shape of x, and the value returned is:
sum(a*weights)/sum(weights)
In computing these sums, elements that correspond to those that are masked in x or weights are ignored.
|
|
choose(condition, t)
has a result shaped like array condition. t must be a tuple of two arrays t1 and t2. Each element of the result is the corresponding element of t1 where condition is true, and the corresponding element of t2 where condition is false. The result is masked where condition is masked or where the selected element is masked.
|
|
concatenate(arrays, axis=0, axisid=None, axisattributes=None)
Concatenate the arrays along the given axis. Give the extended axis the id and attributes provided - by default, those of the first array.
|
|
count(a, axis=None)
Count of the non-masked elements in a, or along a certain axis.
|
|
isMaskedVariable(x)
Return true if x is an instance of a variable.
|
|
masked_equal(x, value)
x masked where x equals the scalar value For floating point value consider masked_values(x, value) instead.
|
|
masked_greater(x, value)
x masked where x > value
|
|
masked_greater_equal(x, value)
x masked where x >= value
|
|
masked_less(x, value)
x masked where x < value
|
|
masked_less_equal(x, value)
x masked where x <= value
|
|
masked_not_equal(x, value)
x masked where x != value
|
|
masked_outside(x, v1, v2)
x with mask of all values of x that are outside [v1,v2]
|
|
masked_where(condition, x, copy=1)
Return x as a variable masked where condition is true. Also masked where x or condition masked. condition is a masked array having the same shape as x.
|
|
maximum(a, b=None)
Compute the maximum valid values of x if y is None; with two arguments, return the element-wise larger of valid values, and mask the result where either x or y is masked.
|
|
minimum(a, b=None)
Compute the minimum valid values of x if y is None; with two arguments, return the element-wise smaller of valid values, and mask the result where either x or y is masked.
|
|
outerproduct(a, b)
Return a variable such that result[i, j] = a[i] * b[j]. The result will be masked where a[i] or b[j] is masked.
|
|
power(a, b)
a**b
|
|
product(a, axis=0, fill_value=1)
Product of elements along axis using fill_value for missing elements.
|
|
repeat(ar, repeats, axis=0)
Return ar repeated repeats times along axis. repeats is a sequence of length ar.shape[axis] telling how many times to repeat each element.
|
|
set_default_fill_value(value_type, value)
Set the default fill value for value_type to value. value_type is a string:
'real','complex','character','integer'
,or
'object'
. value should be a scalar or single-element array.
|
|
sum(a, axis=0, fill_value=0)
Sum of elements along a certain axis using fill_value for missing.
|
|
take(a, indices, axis=0)
Return a selection of items from a. See the documentation in the Numeric manual.
|
|
transpose(ar, axes=None)
Perform a reordering of the axes of array ar depending on the tuple of indices axes;thedefault is to reverse the order of the axes.
|
|
where(condition, x, y)
x where condition is true, y otherwise.
|
Variable
A Variable is a multidimensional data object, consisting of:
-
a multidimensional data array, possibly masked,
-
a collection of attributes
-
a domain, an ordered tuple of Axis and/or Grid objects.
A Variable which is contained in a Dataset or CdmsFile is called a
persistent
variable. Setting a slice of a persistent Variable writes data to the Dataset or file, and referencing a Variable slice reads data from the Dataset. Variables may also be
transient
, not associated with a Dataset or CdmsFile.
Variables support arithmetic operations. The basic Python operators are +,-,*,/,**, abs, and sqrt, together with the operations defined in the MV module. The result of an arithmetic operation is a transient variable, that is, the axis information is transferred to the result.
The methods subRegion and subSlice return transient variables. In ddition, a transient variable may be created with the
cdms.createVariable
method. The vcs and regrid module methods take advantage of the attribute, domain, and mask information in a transient variable.
cu interface support
The cu module defines the original CDAT I/O interface. It is maintained for backward compatibility. As of CDMS V3, CDMS variables support the cu Slab interface defined in cu Module.
Variable Internal Attributes
|
Type
|
Name
|
Definition
|
|
Dictionary
|
attributes
|
External attribute dictionary.
|
|
String
|
id
|
Variable identifier.
|
|
String
|
name_in_file
|
The name of the variable in the file or files which represent the dataset. If different from id, the variable is `aliased'.
|
|
Dataset or CdmsFile
|
parent
|
The dataset or file which contains the variable.
|
|
Tuple
|
shape
|
The length of each axis of the variable.
|
Variable Constructors
|
|
|
Dataset.createVariable(String id, String datatype, List axes)
Create a Variable in a Dataset. This function is not yet implemented.
|
|
CdmsFile.createVariable(String id, String datatype, List axesOrGrids)
Create a Variable in a CdmsFile.
id
is the name of the variable.
datatype
is the MA or Numeric typecode, for example, MA.Float.
axes
OrGrids is a list of Axis and/or Grid objects, on which the variable is defined. Specifying a rectilinear grid is equivalent to listing the grid latitude and longitude axes, in the order defined for the grid. Note: this argument can either be a list or a tuple. If the tuple form is used, and there is only one element, it must have a following comma, e.g.: (axisobj,).
|
|
cdms.createVariable(array, typecode=None, copy=0, savespace=0, mask=None, fill_value=None, grid=None, axes=None, attributes=None, id=None)
Create a transient variable, not associated with a file or dataset.
array
is the data values: a Variable, masked array, or Numeric array.
typecode
is the MA typecode of the array. Defaults to the typecode of
array
.
copy
is an integer flag: if 1, the variable is created with a copy of the array, if 0 the variable data is shared with
array
.
savespace
is an integer flag: if set to 1, internal Numeric operations will attempt to avoid silent upcasting.
mask
is an array of integers with value 0 or 1, having the same shape as
array
.
array
elements with a corresponding mask value of 1 are considered `invalid', and are not used for subsequent Numeric operations. The default mask is obtained from array if present, otherwise is None.
fill_value
is the missing value flag. The default is obtained from
array
if possible, otherwise is set to 1.0e20 for floating point variables, 0 for integer-valued variables.
grid
is a rectilinear grid object.
axes
is a tuple of axis objects. By default the axes are obtained from
array
if present. Otherwise for a dimension of length n, the default axis has values [0., 1., ..., double(n)].
attributes
is a dictionary of attribute values. The dictionary keys must be strings. By default the dictionary is obtained from array if present, otherwise is empty.
id
is the string identifier of the variable. By default the
id
is obtained from array if possible, otherwise is set to 'variable_n' for some integer n.
|
Variable Methods
|
Type
|
Definition
|
|
Variable
|
tvar =
var
[ i:j, m:n]
Read a slice of data from the file or dataset, resulting in a transient variable. Singleton dimensions are `squeezed' out. Data is returned in the physical ordering defined in the dataset. The forms of the slice operator are listed in Table 2.33.
|
|
|
var
[ i:j, m:n] = array
Write a slice of data to the external dataset. The forms of the slice operator are listed in Table 2.21 on page 32. (Variables in CdmsFiles only)
|
|
Variable
|
tvar =
var
(
selector
)
Calling a variable as a function reads the region of data defined by the
selector
. The result is a transient variable, unless
raw=1
keyword is specified. `See Selectors.
|
|
None
|
assignValue(Array ar)
Write the entire data array. Equivalent to var[:] = ar. (Variables in CdmsFiles only).
|
|
Variable
|
astype(typecode)
Cast the variable to a new datatype. Typecodes are as for MV, MA, and Numeric modules.
|
|
Variable
|
clone(copyData=1)
Return a copy of a transient variable.
If copyData is 1 (the default) the variable data is copied as well. If copyData is 0, the result transient variable shares the original transient variable's data array.
|
|
Transient Variable
|
crossSectionRegrid(newLevel, newLatitude, method="log", missing=None, order=None)
Return a lat/level vertical cross-section regridded to a new set of latitudes newLatitude and levels newLevel. The variable should be a function of latitude, level, and (optionally) time.
newLevel is an axis of the result pressure levels.
newLatitude is an axis of the result latitudes.
method is optional, either "
log
" to interpolate in the log of pressure (default), or "
linear
" for linear interpolation.
missing is a missing data value. The default is var.getMissing()
order is an order string such as "tzy" or "zy". The default is var.getOrder()
See also: regrid, pressureRegrid.
|
|
Axis
|
getAxis(n)
Get the n-th axis.
n is an integer.
|
|
List
|
getAxisIds()
Get a list of axis identifiers.
|
|
Integer
|
getAxisIndex(axis_spec)
Return the index of the axis specificed by axis_spec. Return -1 if no match.
axis_spec is a specification as defined for getAxisList
|
|
List
|
getAxisList(axes=None, omit=None, order=None)
Get an ordered list of axis objects in the domain of the variable..
If axes is not None, include only certain axes. Otherwise axes is a list of specifications as described below. Axes are returned in the order specified unless the order keyword is given.
If omit is not None, omit those specified by an integer dimension number. Otherwise omit is a list of specifications as described below.
order is an optional string determining the output order.
Specifications for the axes or omit keywords are a list, each element having one of the following forms:
-
an integer dimension index, starting at 0.
-
a string representing an axis id or one of the strings
'time', 'latitude', 'lat', 'longitude', 'lon', 'lev'
or
'level'
.
-
a function that takes an axis as an argument and returns a value. If the value returned is true, the axis matches.
-
an axis object; will match if it is the same object as axis.
order can be a string containing the characters
t,x,y,z
, or
-
. If a dash ('-') is given, any elements of the result not chosen otherwise are filled in from left to right with remaining candidates.
|
|
List
|
getAxisListIndex(axes=None, omit=None, order=None)
Return a list of indices of axis objects. Arguments are as for getAxisList.
|
|
List
|
getDomain()
Get the domain. Each element of the list is itself a tuple of the form
(axis,start,length,true_length)
where axis is an axis object,
start
is the start index of the domain relative to the axis object,
length
is the length of the axis, and
true_length
is the actual number of (defined) points in the domain.
See also:
getAxisList
.
|
|
Grid
|
getGrid()
Return the associated grid, or None if the variable is not gridded.
|
|
Axis
|
getLatitude()
Get the latitude axis, or None if not found.
|
|
Axis
|
getLevel()
Get the vertical level axis, or None if not found.
|
|
Axis
|
getLongitude()
Get the longitude axis, or None if not found.
|
|
Various
|
getMissing()
Get the missing data value, or None if not found.
|
|
String
|
getOrder()
Get the order string of a spatio-temporal variable. The order string specifies the physical ordering of the data. It is a string of characters with length equal to the rank of the variable, indicating the order of the variable's time, level, latitude, and/or longitude axes. Each character is one of:
't': time
'z': vertical level
'y: latitude
'x': longitude
'-': the axis is not spatio-temporal.
Example: A variable with ordering "tzyx" is 4-dimensional, where the ordering of axes is (time, level, latitude, longitude).
Note: The order string is of the form required for the order argument of a regridder function.
|
|
List
|
getPaths(*intervals)
Get the file paths associated with the index region specified by intervals.
intervals
is a list of scalars, 2-tuples representing [i,j), slices, and/or Ellipses. If no argument(s) are present, all file paths associated with the variable are returned.
Returns a list of tuples of the form (path,slicetuple), where path is the path of a file, and slicetuple is itself a tuple of slices, of the same length as the rank of the variable, representing the portion of the variable in the file corresponding to intervals.
Note: This function is not defined for transient variables.
|
|
String
|
getTemplate()
Get the file template associated with this variable. If no template is associated with the variable, the dataset template is returned.
Note: This function is not defined for transient variables.
|
|
Axis
|
getTime()
Get the time axis, or None if not found.
|
|
Integer
|
len(var)
The length of the first dimension of the variable. If the variable is zero-dimensional (scalar), a length of 0 is returned.
Note:
size()
returns the total number of elements.
|
|
Transient Variable
|
pressureRegrid (newLevel, method="log", missing=None, order=None)
Return the variable regridded to a new set of pressure levels newLevel. The variable must be a function of latitude, longitude, pressure level, and (optionally) time.
newLevel is an axis of the result pressure levels.
method is optional, either "
log
" to interpolate in the log of pressure (default), or "
linear
" for linear interpolation.
missing is a missing data value. The default is var.getMissing()
order is an order string such as "tzyx" or "zyx". The default is var.getOrder()
See also: regrid, crossSectionRegrid.
|
|
Integer
|
rank()
The number of dimensions of the variable.
|
|
Transient Variable
|
regrid (togrid, missing=None, order=None, mask=None)
Return the variable regridded to the horizontal grid togrid.
missing is a Float specifying the missing data value. The default is 1.0e20.
order is a string indicating the order of dimensions of the array. It has the form returned from variable.getOrder(). For example, the string "tzyx" indicates that the dimension order of array is (time, level, latitude, longitude). If unspecified, the function assumes that the last two dimensions of array match the input grid.
mask is a Numeric array, of datatype Integer or Float, consisting of ones and zeros. A value of 0 or 0.0 indicates that the corresponding data value is to be ignored for purposes of regridding. If mask is two-dimensional of the same shape as the input grid, it overrides the mask of the input grid. If the mask has more than two dimensions, it must have the same shape as array. In this case, the missing data value is also ignored. Such an n-dimensional mask is useful if the pattern of missing data varies with level (e.g., ocean data) or time.
Note: If neither
missing
or
mask
is set, the default mask is obtained from the mask of the array if any.
See also: crossSectionRegrid, pressureRegrid.
|
|
None
|
setAxis(n, axis)
Set the n-th axis (0-origin index) of to a copy of axis.
|
|
None
|
setAxisList(axislist)
Set all axes of the variable. axislist is a list of axis objects.
|
|
None
|
setMissing(value)
Set the missing value.
|
|
Integer
|
size()
Number of elements of the variable.
|
|
Variable
|
subRegion(*region, time=None, level=None, latitude=None, longitude=None, squeeze=0, raw=0)
Read a coordinate region of data, returning a transient variable. A region is a hyperrectangle in coordinate space.
region
is an argument list, each item of which specifies an interval of a coordinate axis. The intervals are listed in the order of the variable axes. If trailing dimensions are omitted, all values of those dimensions are retrieved. If an axis is circular (axis.isCircular() is true) or cycle is specified (see below), then data will be read with wraparound in that dimension. Only one axis may be read with wraparound. A coordinate interval has one of the forms listed in Table 2.34. Also see axis.mapIntervalExt.
The optional keyword arguments
time, level, latitude
, and
longitude
may also be used to specify the dimension for which the interval applies. This is particularly useful if the order of dimensions is not known in advance. An exception is raised if a keyword argument conflicts with a positional region argument.
The optional keyword argument
squeeze
determines whether or not the shape of the returned array contains dimensions whose length is 1; by default this argument is 0, and such dimensions are not 'squeezed out'.
The optional keyword argument
raw
specifies whether the return object is a variable or a masked array. By default, a transient variable is returned, having the axes and attributes corresponding to the region read. If
raw
=1, an MA masked array is returned, equivalent to the transient variable without the axis and attribute information.
|
|
Variable
|
subSlice(*specs, time=None, level=None, latitude=None, longitude=None, squeeze=0, raw=0)
Read a slice of data, returning a transient variable. This is a functional form of the slice operator [] with the squeeze option turned off.
specs
is an argument list, each element of which specifies a slice of the corresponding dimension. There can be zero or more positional arguments, each of the form:
(a) a single integer n, meaning slice(n, n+1)
(b) an instance of the slice class
(c) a tuple, which will be used as arguments to create a slice
(d) ':', which means a slice covering that entire dimension
(e) Ellipsis (...), which means to fill the slice list with ':' leaving only enough room at the end for the remaining positional arguments
(f) a Python slice object, of the form slice(i,j,k)
If there are fewer slices than corresponding dimensions, all values of the trailing dimensions are read.
The keyword arguments are defined as in
subRegion
.
There must be no conflict between the positional arguments and the keywords.
In (a)-(c) and (f), negative numbers are treated as offsets from the end of that dimension, as in normal Python indexing.
|
|
String
|
typecode()
The Numeric datatype identifier.
|
Example: Get a region of data.
Variable ta is a function of (time, latitude, longitude). Read data corresponding to all times, latitudes -45.0 up to but not including +45.0, longitudes 0.0 through and including longitude 180.0:
data = ta.subRegion(':', (-45.0,45.0,'co'), (0.0, 180.0))
or equivalently:
data = ta.subRegion(latitude=(-45.0,45.0,'co'), longitude=(0.0, 180.0)
Read all data for March, 1980:
data = ta.subRegion(time=('1980-3','1980-4','co'))
Variable Slice Operators
|
[i]
|
The ith element, zero-origin indexing.
|
|
[i:j]
|
The ith element through, but not including, element j
|
|
[i:]
|
The ith element through the end
|
|
[:j]
|
The beginning element through, but not including, element j
|
|
[:]
|
The entire array
|
|
[i:j:k]
|
Every kth element
|
|
[i:j, m:n]
|
Multidimensional slice
|
|
[i, ..., m]
|
(Ellipsis) All dimensions between those specified.
|
|
[-1]
|
Negative indices 'wrap around'. -1 is the last element.
|
Index and Coordinate Intervals
|
Interval
|
Definition
|
Example
|
|
x
|
single point, such that axis[i]==x
In general x is a scalar. If the axis is a time axis, x may also be a cdtime relative time type, component time type, or string of the form 'yyyy-mm-dd hh:mi:ss' (where trailing fields of the string may be omitted.
|
180.0
cdtime.reltime(48,"hours since 1980-1")
'1980-1-3'
|
|
(x,y)
|
indices i such that x <= axis[i] <= y
|
(-180,180)
|
|
(x,y,'co')
|
x <= axis[i] < y
The third item is defined as in mapInterval.
|
(-90,90,'cc')
|
|
(x,y,'co',cycle)
|
x<= axis[i] < y, with wraparound
Note: It is not necesary to specify the cycle of a circular longitude axis, that is, for which axis.isCircular() is true.
|
( 180, 180, 'co', 360.0)
|
|
slice
(i,j,k)
|
slice object, equivalent to i:j:k in a slice operator. Refers to the indices i, i+k, i+2k, ... up to but not including index j. If i is not specified or is None it defaults to 0. If j is not specified or is None it defaults to the length of the axis. The stride k defaults to 1. k may be negative.
|
slice(1,10)
slice(,,-1) reverses the direction of the axis.
|
|
':'
|
all axis values of one dimension
|
|
|
Ellipsis
|
all values of all intermediate axes
|
|
Selectors
A
selector
is a specification of a region of data to be selected from a variable. For example, the statement
x = v(time='1979-1-1', level=(1000.0,100.0))
means `select the values of variable v for time '1979-1-1' and levels 1000.0 to 100.0 inclusive, setting x to the result.' Selectors are generally used to represent regions of space and time.
The form for using a selector is
result = v(s)
where
v
is a variable and
s
is the selector. An equivalent form is
result = f(`varid', s)
where
f
is a file or dataset, and
`varid'
is the string ID of a variable.
A selector consists of a list of
selector components
. For example, the selector
time='1979-1-1', level=(1000.0,100.0)
has two components:
time='1979-1-1'
, and
level=(1000.0,100.0)
. This illustrates that selector components can be defined with keywords, using the form:
keyword=value
Note that for the keywords t
ime, level, latitude
, and
longitude
, the selector can be used with any variable. If the corresponding axis is not found, the selector component is ignored. This is very useful for writing general purpose scripts. The
required
keyword overrides this behavior. These keywords take values that are coordinate ranges or index ranges as defined in Table 2.34.
The following keywords are available:
Selector keywords
|
Keyword
|
Description
|
Value
|
|
axisid
|
Restrict the axis with ID
axisid
to a value or range of values.
|
See Table 2.34
|
|
grid
|
Regrid the result to the grid.
|
Grid object
|
|
latitude
|
Restrict latitude values to a value or range. Short form: lat
|
See Table 2.34
|
|
level
|
Restrict vertical levels to a value or range. Short form: lev
|
See Table 2.34
|
|
longitude
|
Restrict longitude values to a value or range. Short form: lon
|
See Table 2.34
|
|
order
|
Reorder the result.
|
Order string, e.g., "tzyx"
|
|
raw
|
Return a masked array (MA.array) rather than a transient variable.
|
0: return a transient variable (default); =1: return a masked array.
|
|
required
|
Require that the axis IDs be present.
|
List of axis identifiers.
|
|
squeeze
|
Remove singleton dimensions from the result.
|
0: leave singleton dimensions (default); 1: remove singleton dimensions.
|
|
time
|
Restrict time values to a value or range.
|
See Table 2.34
|
Another form of selector components is the
positional
form, where the component order corresponds to the axis order of a variable. For example:
x9 = hus(('1979-1-1','1979-2-1'),1000.0)
reads data for the range
('1979-1-1','1979-2-1')
of the first axis, and coordinate value
1000.0
of the second axis. Non-keyword arguments of the form(s) listed in Table 2.34 are treated as positional. Such selectors are more concise, but not as general or flexible as the other types described in this section.
Selectors are objects in their own right. This means that a selector can be defined and reused, independent of a particular variable. Selectors are constructed using the cdms.selectors.Selector class. The constructor takes an argument list of selector components. For example:
from cdms.selectors import Selector
sel = Selector(time=('1979-1-1','1979-2-1'), level=1000.)
x1 = v1(sel)
x2 = v2(sel)
For convenience CDMS provides several predefined selectors, which can be used directly or can be combined into more complex selectors. The selectors
time, level, latitude, longitude
, and
required
are equivalent to their keyword counterparts. For example:
from cdms import time, level
x = hus(time('1979-1-1','1979-2-1'), level(1000.))
and
x = hus(time=('1979-1-1','1979-2-1'), level=1000.)
are equivalent. Additionally, the predefined selectors
latitudeslice, longitudeslice, levelslice
, and
timeslice
take arguments (startindex, stopindex[, stride]):
from cdms import timeslice, levelslice
x = v(timeslice(0,2), levelslice(16,17))
Finally, a collection of selectors is defined in module
cdutil.region
:
from cdutil.region import *
NH=NorthernHemisphere=domain(latitude=(0.,90.)
SH=SouthernHemisphere=domain(latitude=(-90.,0.))
Tropics=domain(latitude=(-23.4,23.4))
NPZ=AZ=ArcticZone=domain(latitude=(66.6,90.))
SPZ=AAZ=AntarcticZone=domain(latitude=(-90.,-66.6))
Selectors can be combined using the & operator, or by refining them in the call:
from cdms.selectors import Selector
from cdms import level
sel2 = Selector(time=('1979-1-1','1979-2-1'))
sel3 = sel2 & level(1000.0)
x1 = hus(sel3)
x2 = hus(sel2, level=1000.0)
Selector examples
CDMS provides a variety of ways to select or slice data. In the following examples, variable `hus' is contained in file sample.nc, and is a function of (time, level, latitude, longitude). Time values are monthly starting at 1979-1-1. There are 17 levels, the last level being 1000.0. The name of the vertical level axis is `plev'. All the examples select the first two times and the last level. The last two examples remove the singleton level dimension from the result array.
import cdms
f = cdms.open('sample.nc')
hus = f.variables['hus']
# Keyword selection
x = hus(time=('1979-1-1','1979-2-1'), level=1000.)
# Interval indicator (see mapIntervalExt)
x = hus(time=('1979-1-1','1979-3-1','co'), level=1000.)
# Axis ID (plev) as a keyword
x = hus(time=('1979-1-1','1979-2-1'), plev=1000.)
# Positional
x9 = hus(('1979-1-1','1979-2-1'),1000.0)
# Predefined selectors
from cdms import time, level
x = hus(time('1979-1-1','1979-2-1'), level(1000.))
from cdms import timeslice, levelslice
x = hus(timeslice(0,2), levelslice(16,17))
# Call file as a function
x = f('hus', time=('1979-1-1','1979-2-1'), level=1000.)
# Python slices
x = hus(time=slice(0,2), level=slice(16,17))
# Selector objects
from cdms.selectors import Selector
sel = Selector(time=('1979-1-1','1979-2-1'), level=1000.)
x = hus(sel)
sel2 = Selector(time=('1979-1-1','1979-2-1'))
sel3 = sel2 & level(1000.0)
x = hus(sel3)
x = hus(sel2, level=1000.0)
# Squeeze singleton dimension (level)
x = hus[0:2,16]
x = hus(time=('1979-1-1','1979-2-1'), level=1000., squeeze=1)
f.close()
Examples
In this example, two datasets are opened, containing surface air temperature (`tas') and upper-air temperature (`ta') respectively. Surface air temperature is a function of (time, latitude, longitude). Upper-air temperature is a function of (time, level, latitude, longitude). Time is assumed to have a relative representation in the datasets (e.g., with units "months since basetime").
Data is extracted from both datasets for January of the first input year through December of the second input year. For each time and level, three quantities are calculated: slope, variance, and correlation. The results are written to a netCDF file. For brevity, the functions corrCoefSlope and removeSeasonalCycle are omitted.
import cdms
from cdms import MV
# Calculate variance, slope, and correlation of
# surface air temperature with upper air temperature
# by level, and save to a netCDF file. 'pathTa' is the location of
# the file containing ta, 'pathTas' is the file with contains tas.
# Data is extracted from January of year1 through December of year2.
def ccSlopeVarianceBySeasonFiltNet(pathTa,pathTas,month1,month2):
# Open the files for ta and tas
fta = cdms.open(pathTa)
ftas = cdms.open(pathTas)
# Get upper air temperature
taObj = fta['ta']
levs = taObj.getLevel()
# Get the surface temperature for the closed interval [time1,time2]
tas = ftas('tas', time=(month1,month2,'cc'))
# Allocate result arrays
newaxes = taObj.getAxisList(omit='time')
newshape = tuple([len(a) for a in newaxes])
cc = MV.zeros(newshape, typecode=MV.Float, axes=newaxes, id='correlation')
b = MV.zeros(newshape, typecode=MV.Float, axes=newaxes, id='slope')
v = MV.zeros(newshape, typecode=MV.Float, axes=newaxes, id='variance')
# Remove seasonal cycle from surface air temperature
tas = removeSeasonalCycle(tas)
# For each level of air temperature, remove seasonal cycle
# from upper air temperature, and calculate statistics
for ilev in range(len(levs)):
ta = taObj.subRegion(time=(month1,month2,'cc'), \
level=slice(ilev, ilev+1), squeeze=1)
ta = removeSeasonalCycle(ta)
cc[ilev], b[ilev] = corrCoefSlope(tas ,ta)
v[ilev] = MV.sum( ta**2 )/(1.0*ta.shape[0])
# Write slope, correlation, and variance variables
f = cdms.open('CC_B_V_ALL.nc','w')
f.title = 'filtered'
f.write(b)
f.write(cc)
f.write(v)
f.close()
if __name__=='__main__':
pathTa = '/pcmdi/cdms/sample/ccmSample_ta.xml'
pathTas = '/pcmdi/cdms/sample/ccmSample_tas.xml'
# Process Jan80 through Dec81
ccSlopeVarianceBySeasonFiltNet(pathTa,pathTas,'80-1','81-12')
Notes:
-
Two modules are imported, cdms, and MV. MV implements arithmetic functions.
-
taObj
is a file (persistent) variable. At this point, no data has actually been read. This happens when the file variable is sliced, or when the subRegion function is called.
levs
is an axis.
-
Calling the file like a function reads data for the given variable and time range. Note that
month1
and
month2
are time strings.
-
In contrast to
taObj
, the variables
cc, b
, and
v
are transient variables, not associated with a file. The assigned names are used when the variables are written.
-
Another way to read data is to call subRegion on the variable. The squeeze option removes singleton axes, in this case the level axis.
-
Write the data. Axis information is written automatically.
-
This is the main routine of the script.
pathTa
and
pathTas
pathnames. Data is processed from January 1980 through December 1981.
In the next example, the pointwise variance of a variable over time is calculated, for all times in a dataset. The name of the dataset and variable are entered, then the variance is calculated and plotted via the vcs module.
#!/usr/bin/env python
#
# Calculates gridpoint total variance
# from an array of interest
#
import cdms
from MV import *
# Wait for return in an interactive window
def pause():
print 'Hit return to continue: ',
line = sys.stdin.readline()
# Calculate pointwise variance of variable over time
# Returns the variance and the number of points
# for which the data is defined, for each grid point
def calcVar(x):
# Check that the first axis is a time axis
firstaxis = x.getAxis(0)
if not firstaxis.isTime():
raise 'First axis is not time, variable:', x.id
n = count(x,0)
sumxx = sum(x*x)
sumx = sum(x)
variance = (n*sumxx - (sumx * sumx))/(n * (n-1.))
return variance,n
if __name__=='__main__':
import vcs, sys
print 'Enter dataset path [/pcmdi/cdms/obs/erbs_mo.xml]: ',
path = string.strip(sys.stdin.readline())
if path=='': path='/pcmdi/cdms/obs/erbs_mo.xml'
# Open the dataset
dataset = cdms.open(path)
# Select a variable from the dataset
print 'Variables in file:',path
varnames = dataset.variables.keys()
varnames.sort()
for varname in varnames:
var = dataset.variables[varname]
if hasattr(var,'long_name'):
long_name = var.long_name
elif hasattr(var,'title'):
long_name = var.title
else:
long_name = '?'
print '%-10s: %s'%(varname,long_name)
print 'Select a variable: ',
varname = string.strip(sys.stdin.readline())
var = dataset(varname)
dataset.close()
# Calculate variance, count, and set attributes
variance,n = calcVar(var)
variance.id = 'variance_%s'%var.id
n.id = 'count_%s'%var.id
if hasattr(var,'units'):
variance.units = '(%s)^2'%var.units
# Plot variance
w=vcs.init()
w.plot(variance)
pause()
w.clear()
w.plot(n)
pause()
w.clear()
The result of running this script is as follows:
% calcVar.py
Enter dataset path [/pcmdi/cdms/sample/obs/erbs_mo.xml]:
Variables in file: /pcmdi/cdms/sample/obs/erbs_mo.xml
albt : Albedo TOA [%]
albtcs : Albedo TOA clear sky [%]
rlcrft : LW Cloud Radiation Forcing TOA [W/m^2]
rlut : LW radiation TOA (OLR) [W/m^2]
rlutcs : LW radiation upward TOA clear sky [W/m^2]
rscrft : SW Cloud Radiation Forcing TOA [W/m^2]
rsdt : SW radiation downward TOA [W/m^2]
rsut : SW radiation upward TOA [W/m^2]
rsutcs : SW radiation upward TOA clear sky [W/m^2]
Select a variable: albt
<The variance is plotted>
Hit return to continue:
<The number of points is plotted>
Notes:
-
n = count(x, 0)
returns the pointwise number of valid values, summing across axis 0, the first axis.
count
is an MV function.
-
dataset
is a Dataset or CdmsFile object, depending on whether a .xml or .nc pathname is entered.
dataset.variables
is a dictionary mapping variable name to file variable.
-
var
is a transient variable.
-
Plot the variance and count variables. Spatial longitude and latitude information are carried with the computations, so the continents are plotted correctly.
Overview
