A first example
The following Python script reads January and July monthly temperature data from an input dataset, averages over time, and writes the results to an output file. The input temperature data is ordered (time, latitude, longitude).
1 #!/usr/bin/env python
2 import cdms
3 from cdms import MV
4 jones = cdms.open('/pcmdi/cdms/obs/jones_mo.nc')
5 tasvar = jones['tas']
6 jans = tasvar[0::12]
7 julys = tasvar[6::12]
8 janavg = MV.average(jans)
9 janavg.id = "tas_jan"
10 janavg.long_name = "mean January surface temperature"
11 julyavg = MV.average(julys)
12 julyavg.id = "tas_jul"
13 julyavg.long_name = "mean July surface temperature"
14 out = cdms.open('janjuly.nc','w')
15 out.write(janavg)
16 out.write(julyavg)
17 out.comment = "Average January/July from Jones dataset"
18 jones.close()
19 out.close()
|
Line
|
Notes
|
|
2,3
|
Makes the CDMS and MV modules available. MV defines arithmetic functions.
|
|
4
|
Opens a netCDF file read-only. The result
jones
is a dataset object.
|
|
5
|
Gets the surface air temperature variable.
'tas'
is the name of the variable in the input dataset. This does not actually read the data.
|
|
6
|
Read all January monthly mean data into a variable
jans
. Variables can be sliced like arrays. The slice operator
[0::12]
means `take every 12th slice from dimension 0, starting at index 0 and ending at the last index.' If the stride
12
were omitted, it would default to 1.
Note that the variable is actually 3-dimensional. Since no slice is specified for the second or third dimensions, all values of those dimensions are retrieved. The slice operation could also have been written
[0::12, : , :]
.
Also note that the same script works for multi-file datasets. CDMS opens the needed data files, extracts the appropriate slices, and concatenates them into the result array.
|
|
7
|
Reads all July data into a masked array
julys
.
|
|
8
|
Calculate the average January value for each grid zone. Any missing data is handled automatically.
|
|
9,10
|
Set the variable
id
and
long_name
attributes. The id is used as the name of the variable when plotted or written to a file.
|
|
14
|
Create a new netCDF output file named
'janjuly.nc'
to hold the results.
|
|
15
|
Write the January average values to the output file. The variable will have id "
tas_jan
" in the file.
write
is a utility function which creates the variable in the file, then writes data to the variable. A more general method of data output is first to create a variable, then set a slice of the variable.
Note that
janavg
and
julavg
have the same latitude and longitude information as
tasvar
. It is carried along with the computations.
|
|
17
|
Set the global attribute '
comment
'.
|
|
18
|
Close the output file.
|
CdmsObj
A CdmsObj is the base class for all CDMS database objects. At the application level, CdmsObj objects are never created and used directly. Rather the subclasses of CdmsObj (Dataset, Variable, Axis, etc.) are the basis of user application programming.
All objects derived from CdmsObj have a special attribute .attributes. This is a Python dictionary, which contains all the external (persistent) attributes associated with the object. This is in contrast to the internal, non-persistent attributes of an object, which are built-in and predefined. When a CDMS object is written to a file, the external attributes are written, but not the internal attributes.
Example: get a list of all external attributes of obj.
extatts = obj.attributes.keys()
Attributes common to all CDMS objects
|
Type
|
Name
|
Definition
|
|
Dictionary
|
attributes
|
External attribute dictionary for this object.
|
All attributes may be accessed and set using the Python dot notation (`.')
Getting and setting attributes
|
Type
|
Definition
|
|
Various
|
value = obj.attname
Get an internal or external attribute value. If the attribute is external, it is read from the database. If the attribute is not already in the database, it is created as an external attribute. Internal attributes cannot be created, only referenced.
|
|
|
obj.attname = value
Set an internal or external attribute value. If the attribute is external, it is written to the database.
|
Database
A Database is a collection of datasets and other CDMS objects. It consists of a hierarchical collection of objects, with the database being at the root, or top of the hierarchy. A database is used to:
-
search for metadata
-
access data
-
provide authentication and access control for data and metadata
The figure below illustrates several important points:
-
Each object in the database has a relative name of the form tag=id. The id of an object is unique with respect to all objects contained in the parent.
-
The name of the object consists of its relative name followed by the relative name(s) of its antecedent objects, up to and including the database name. In the figure below, one of the variables has name
"variable=ua, dataset=ncep_reanalysis_mo,database=CDMS".
-
Subordinate objects are thought of as being contained in the parent. In this example, the database `CDMS' contains two datasets, each of which contain several variables.
Overview
To access a database:
-
Open a connection. The connect method opens a database connection. connect takes a database URI and returns a database object:
db = cdms.connect("ldap://dbhost.llnl.gov/database=CDMS,ou=PCMDI,o=LLNL,c=US")
-
Search the database, locating one or more datasets, variables, and/or other objects.
The database searchFilter method searches the database. A single database connection may be used for an arbitrary number of searches.
For example, to find all observed datasets:
result = db.searchFilter("category=observed",tag="dataset")
Searches can be restricted to a subhierarchy of the database. This example searches just the dataset `ncep_reanalysis_mo':
result = db.searchFilter(relbase="dataset=ncep_reanalysis")
-
Refine the search results if necessary. The result of a search can be narrowed with the searchPredicate method.
-
Process the results. A search result consists of a sequence of entries. Each entry has a name, the name of the CDMS object, and an attribute dictionary, consisting of the attributes located by the search:
for entry in result:
print entry.name, entry.attributes
-
Access the data. The CDMS object associated with an entry is obtained from the getObject method:
obj = entry.getObject()
If the id of a dataset is known, the dataset can be opened directly with the open method:
dset = db.open("ncep_reanalysis_mo")
-
Close the database connection:
db.close()
Database Internal Attributes
|
Type
|
Name
|
Summary
|
|
Dictionary
|
attributes
|
Database attribute dictionary
|
|
LDAP
|
db
|
(LDAP only) LDAP database object
|
|
String
|
netloc
|
Hostname, for server-based databases
|
|
String
|
path
|
path name
|
|
String
|
uri
|
Uniform Resource Identifier.
|
Database Constructors
|
|
|
db = cdms.connect(uri=None, user="", password="")
Connect to the database.
uri is the Universal Resource Indentifier of the database. The form of the URI depends on the implementation of the database. For a Lightweight Directory Access Protocol (LDAP) database, the form is:
ldap://host[:port]/dbname
For example, if the database is located on host `dbhost.llnl.gov', and is named 'database=CDMS,ou=PCMDI,o=LLNL,c=US', the URI is:
ldap://dbhost.llnl.gov/database=CDMS,ou=PCMDI,o=LLNL,c=US
If unspecified, the URI defaults to the value of environment variable CDMSROOT.
user is the user ID. If unspecified, an anonymous connection is made.
password is the user password. A password is not required for an anonymous connection.
|
Database Methods
|
Type
|
Definition
|
|
None
|
close()
Close a database connection.
|
|
List
|
listDatasets()
Return a list of the dataset IDs in this database. A dataset ID can be passed to the
open
command.
|
|
Dataset
|
open(dsetid, mode='r')
Open a dataset.
dsetid is the string dataset identifier
mode is the open mode, 'r' - read-only, 'r+' - read-write, 'w' - create.
openDataset
is a synonym for
open
.
|
|
SearchResult
|
searchFilter(filter=None, tag=None, relbase=None, scope=Subtree, attnames=None, timeout=None)
Search a CDMS database.
filter is the string search filter. Simple filters have the form "tag = value". Simple filters can be combined using logical operators '&', '|', '!' in prefix notation. For example, the filter '(&(objectclass=variable)(id=cli))' finds all variables named "cli". A formal definition of search filters is provided in the following section.
tag restricts the search to objects with that tag ("dataset" | "variable" | "database" | "axis" | "grid").
relbase is the relative name of the base object of the search. The search is restricted to the base object and all objects below it in the hierarchy. For example, to search only dataset `ncep_reanalysis_mo', specify:
relbase="dataset=ncep_reanalysis_mo".
To search only variable 'ua' in ncep_reanalysis_mo, use:
relbase="variable=ua, dataset=ncep_reanalysis_mo"
If no base is specified, the entire database is searched. See the scope argument also.
scope is the search scope (Subtree | Onelevel | Base). Subtree searches the base object and its descendants. Onelevel searches the base object and its immediate descendants. Base searches the base object alone. The default is Subtree.
attnames: list of attribute names. Restricts the attributes returned. If None, all attributes are returned. Attributes 'id' and 'objectclass' are always included in the list.
timeout: integer number of seconds before timeout. The default is no timeout.
|
Searching a database
The searchFilter method is used to search a database. The result is called a search result, and consists of a sequence of result entries.
In its simplest form, searchFilter takes an argument consisting of a string filter. The search returns a sequence of entries, corresponding to those objects having an attribute which matches the filter. Simple filters have the form (tag = value), where value can contain wildcards. For example:
'(id = ncep*)'
'(project = AMIP2)'
Simple filters can be combined with the logical operators `&', `|', `!'. For example,
'(&(id = bmrc*)(project = AMIP2))'
matches all objects with id starting with 'bmrc', and a 'project' attribute with value 'AMIP2'.
Formally, search filters are strings defined as follows:
filter ::= "(" filtercomp ")"
filtercomp ::= "&" filterlist | # and
"|" filterlist | # or
"!" filterlist | # not
simple
filterlist ::= filter | filter filterlist
simple ::= tag op value
op ::= "=" | # equality
"~=" | # approximate equality
"<=" | # lexicographically less than or equal to
">=" # lexicographically greater than or equal to
tag ::= string attribute name
value ::= string attribute value, may include '*' as a wild card
Attribute names are defined in the chapter on Climate Data Markup Language (CDML). In addition, some special attributes are defined for convenience:
-
category is either "experimental" or "observed"
-
parentid is the ID of the parent dataset
-
project is a project identifier, e.g., "AMIP2"
-
objectclass is the list of tags associated with the object.
The set of objects searched is called the search scope. The top object in the hierarchy is the base object. By default, all objects in the database are searched, that is, the database is the base object. If the database is very large, this may result in an unnecessarily slow or inefficient search. To remedy this the search scope can be limited in several ways:
-
The base object can be changed.
-
The scope can be limited to the base object and one level below, or to just the base object.
-
The search can be restricted to objects of a given class (dataset, variable, etc.)
-
The search can be restricted to return only a subset of the object attributes
-
The search can be restricted to the result of a previous search.
A search result is accessed sequentially within a for loop:
result = db.searchFilter('(&(category=obs*)(id=ncep*))')
for entry in result:
print entry.name
Search results can be narrowed using searchPredicate. In the following example, the result of one search is itself searched for all variables defined on a 94x192 grid:
>>> result = db.searchFilter('parentid=ncep*',tag="variable")
>>> len(result)
65
>>> result2 = result.searchPredicate(lambda x: x.getGrid().shape==(94,192))
>>> len(result2)
3
>>> for entry in result2: print entry.name
variable=rluscs,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
variable=rlds,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
variable=rlus,dataset=ncep_reanalysis_mo,database=CDMS,ou=PCMDI, o=LLNL, c=US
>>>
SearchResult Methods
|
Type
|
Definition
|
|
ResultEntry
|
[i]
Return the i-th search result. Results can also be returned in a for loop:
for entry in db.searchResult(tag="dataset"):
...
|
|
Integer
|
len()
Number of entries in the result.
|
|
SearchResult
|
searchPredicate(predicate, tag=None)
Refine a search result, with a predicate search.
predicate is a function which takes a single CDMS object and returns true (1) if the object satisfies the predicate, 0 if not.
tag restricts the search to objects of the class denoted by the tag.
Note: In the current implementation, searchPredicate is much less efficient than searchFilter. For best performance, use searchFilter to narrow the scope of the search, then use searchPredicate for more general searches.
|
A search result is a sequence of result entries. Each entry has a string name, the name of the object in the database hierarchy, and an attribute dictionary. An entry corresponds to an object found by the search, but differs from the object, in that only the attributes requested are associated with the entry. In general, there will be much more information defined for the associated CDMS object, which is retrieved with the getObject method.
ResultEntry Attributes
|
Type
|
Name
|
Summary
|
|
String
|
name
|
The name of this entry in the database.
|
|
Dictionary
|
attributes
|
The attributes returned from the search.
attributes[key] is a list of all string values associated with the key.
|
ResultEntry Methods
|
Type
|
Definition
|
|
CdmsObj
|
getObject()
Return the CDMS object associated with this entry.
Note: For many search applications it is unnecessary to access the associated CDMS object. For best performance this function should be used only when necessary, for example, to retrieve data associated with a variable.
|
Accessing data
To access data via CDMS:
-
Locate the dataset ID. This may involve searching the metadata.
-
Open the dataset, using the open method.
-
Reference the portion of the variable to be read.
In the next example, a portion of variable 'ua' is read from dataset 'ncep_reanalysis_mo':
dset = db.open('ncep_reanalysis_mo')
ua = dset.variables['ua']
data = ua[0,0]
Examples of database searches
In the following examples, db is the database opened with
db = cdms.connect()
This defaults to the database defined in environment variable CDMSROOT.
List all variables in dataset 'ncep_reanalysis_mo':
for entry in db.searchFilter(filter="parentid=ncep_reanalysis_mo", tag="variable"):
print entry.name
Find all axes with bounds defined:
for entry in db.searchFilter(filter="bounds=*",tag="axis"):
print entry.name
Locate all GDT datasets:
for entry in db.searchFilter(filter="Conventions=GDT*",tag="dataset"):
print entry.name
Find all variables with missing time values, in observed datasets:
def missingTime(obj):
time = obj.getTime()
return time.length != time.partition_length
result = db.searchFilter(filter="category=observed")
for entry in result.searchPredicate(missingTime):
print entry.name
Find all CMIP2 datasets having a variable with id "hfss":
for entry in db.searchFilter(filter="(&(project=CMIP2)(id=hfss))",tag="variable"):
print entry.getObject().parent.id
Find all observed variables on 73x144 grids:
result = db.searchFilter('category=obs*')
for entry in result.searchPredicate(lambda x: x.getGrid().shape==(73,144),tag="variable"):
print entry.name
Find all observed variables with more than 1000 timepoints:
result = db.searchFilter('category=obs*')
for entry in result.searchPredicate(lambda x: len(x.getTime())>1000, tag="variable"):
print entry.name, len(entry.getObject().getTime())
Find the total number of each type of object in the database
print len(db.searchFilter(tag="database")),"database"
print len(db.searchFilter(tag="dataset")),"datasets"
print len(db.searchFilter(tag="variable")),"variables"
print len(db.searchFilter(tag="axis")),"axes"
Dataset
A Dataset is a virtual file. It consists of a metafile, in CDML/XML representation, and one or more data files.
As of CDMS V3, the legacy cuDataset interface is supported by Datasets. See cu Module.
Dataset Internal Attributes
|
Type
|
Name
|
Summary
|
|
Dictionary
|
attributes
|
Dataset external attributes.
|
|
Dictionary
|
axes
|
Axes contained in the dataset.
|
|
String
|
datapath
|
Path of data files, relative to the parent database. If no parent, the datapath is absolute.
|
|
Dictionary
|
grids
|
Grids contained in the dataset.
|
|
String
|
mode
|
Open mode.
|
|
Database
|
parent
|
Database which contains this dataset. If the dataset is not part of a database, the value is None.
|
|
String
|
uri
|
Uniform Resource Identifier of this dataset.
|
|
Dictionary
|
variables
|
Variables contained in the dataset.
|
|
Dictionary
|
xlinks
|
External links contained in the dataset.
|
Dataset Constructors
|
|
|
datasetobj
= cdms.open(String uri, String mode='r')
Open the dataset specified by the Universal Resource Indicator, a CDML file. Returns a Dataset object. mode is one of the indicators listed in Table 2.22.
openDataset
is a synonym for
open
.
|
|
datasetobj
= cdms.createDataset(String path, String directory, String fileTemplate)
(Note: this function is not yet implemented)
Create a new dataset, returning a Dataset object.
path
is the filepath of a CDML file.
fileTemplate
describes how the dataset is to be partitioned. It is a pathname, relative to the directory, which contains zero or more template specifiers (see Table 2.23). A template may contain directory names as well as file names. A template specifier is a string of the form '%X' or '%eX', where X is one of the characters listed Table 2.23. The form '%eX' may be used to specify the end time or level value. A specifier may appear more than once in a template.
|
Open Modes
|
Mode
|
Definition
|
|
'r'
|
read-only
|
|
'r+'
|
read-write
|
|
'a'
|
read-write. Open the file if it exists, otherwise create a new file
|
|
'w'
|
Create a new file, read-write
|
Template Specifiers
|
Specifier
|
Definition
|
Example
|
|
d
|
day number
|
1 .. 31
|
|
f
|
day number, two-digit, zero-filled
|
01 .. 31
|
|
g
|
month, lower case, three characters
|
'jan', 'feb', ...
|
|
G
|
month, upper case, three characters
|
'JAN', 'FEB', ...
|
|
H
|
hour
|
0 .. 23
|
|
L
|
vertical level
|
integer
|
|
m
|
month number, not zero filled
|
1 .. 12
|
|
M
|
minute
|
0 .. 59
|
|
n
|
month number, two-digit, zero-filled
|
01, 02, ..., 12
|
|
S
|
second
|
0 .. 59
|
|
v
|
variable ID
|
character
|
|
y
|
year, two-digit, zero-filled
|
integer
|
|
Y
|
year
|
integer
|
|
z
|
Zulu time
|
ex: '6Z19990201'
|
|
%
|
percent sign
|
'%'
|
For example, the file template
ccsr-a/mo/%v/ccsr-a/%v_ccsr-a_%Y.%n-%eY.%en.nc
contains the specifiers %v (variable name), %Y (year), %eY (end year), and %en (end month). One of the files in the dataset might have the path (relative to the parent directory)
ccsr-a/mo/ta/ccsr-a/ta_ccsr-a_1979.01-1979.12.nc
Dataset Methods
|
Type
|
Definition
|
|
TransientVariable
|
datasetobj
(varname, selector)
Calling a Dataset object as a function reads the region of data defined by the selector. The result is a transient variable, unless
raw=1
is specified. See Selectors.
For example, the following reads data for variable 'prc', year 1980:
f = cdms.open('test.xml')
x = f('prc', time=('1980-1','1981-1'))
|
|
Variable, Axis, or Grid
|
datasetobj
['id']
The square bracket operator applied to a dataset gets the persistent variable, axis or grid object having the string identifier. This does not read the data for a variable. Returns None if not found.
For example:
f = cdms.open('sample.xml')
v = f['prc']
gets the persistent variable v, equivalent to
v=f.variables['prc']
.
t = f['time']
gets the axis named `time', equivalent to
t=f.axes['time']
.
|
|
None
|
close()
Close the dataset.
|
|
RectGrid
|
createRectGrid(id, lat, lon, order, type="generic", mask=None)
Create a RectGrid in the dataset. This is not a persistent object: the order, type, and mask are not written to the dataset. However, the grid may be used for regridding operations.
lat is a latitude axis in the dataset.
lon is a longitude axis in the dataset.
order is a string with value "yx" (the first grid dimension is latitude) or "xy" (the first grid dimension is longitude).
type is one of 'gaussian','uniform','equalarea',or 'generic'
If specified, mask is a two-dimensional, logical Numeric array (all values are zero or one) with the same shape as the grid.
|
|
Axis
|
getAxis(id)
Get an axis object from the file or dataset.
id
is the string axis identifier.
|
|
Grid
|
getGrid(id)
Get a grid object from a file or dataset.
id
is the string grid identifier.
|
|
List
|
getPaths()
Get a sorted list of pathnames of datafiles which comprise the dataset. This does not include the XML metafile path, which is stored in the .uri attribute.
|
|
Variable
|
getVariable(id)
Get a variable object from a file or dataset.
id
is the string variable identifier.
|
|
None
|
sync()
Write any pending changes to the dataset.
|
MV module
The fundamental CDMS data object is the variable. A variable is comprised of:
-
a masked data array, as defined in the NumPy MA module.
-
a domain: an ordered list of axes and/or grids.
-
an attribute dictionary.
The MV module is a work-alike replacement for the MA module, that carries along the domain and attribute information where appropriate. MV provides the same set of functions as MA. However, MV functions generate transient variables as results. Often this simplifies scripts that perform computation. MA is part of the Python Numeric package, documented at http://numpy.sourceforge.net.
MV can be imported with the command:
import MV
The command
from MV import *
allows use of MV commands without any prefix.
Table 2.25 lists the constructors in MV. All functions return a transient variable. In most cases the keywords axes, attributes, and id are available. axes is a list of axis objects which specifies the domain of the variable. attributes is a dictionary. id is a special attribute string that serves as the identifier of the variable, and should not contain blanks or non-printing characters. It is used when the variable is plotted or written to a file. Since the id is just an attribute, it can also be set like any attribute:
var.id = 'temperature'
For completeness MV provides access to all the MA functions. The functions not listed in the following tables are identical to the corresponding MA function: allclose, allequal, common_fill_value, compress, create_mask, dot, e, fill_value, filled, get_print_limit, getmask, getmaskarray, identity, indices, innerproduct, isMA, isMaskedArray, is_mask, isarray, make_mask, make_mask_none, mask_or, masked, pi, put, putmask, rank, ravel, set_fill_value, set_print_limit, shape, size, sort. See the documentation at http://numpy.sourceforge.net for a description of these functions.
Variable Constructors in module MV
|
|
|
arrayrange(start, stop=None, step=1, typecode=None, axis=None, attributes=None, id=None)
Just like MA.arange() except it returns a variable whose type can be specfied by the keyword argument typecode. The axis, attribute dictionary, and string identifier of the result variable may be specified.
Synonym: arange
|
|
masked_array(a, mask=None, fill_value=None, axes=None, attributes=None, id=None)
Same as MA.masked_array but creates a variable instead. If no axes are specified, the result has default axes, otherwise axes is a list of axis objects matching a.shape.
|
|
masked_object(data, value, copy=1, savespace=0, axes=None, attributes=None, id=None)
Create variable masked where exactly data equal to value. Create the variable with the given list of axis objects, attribute dictionary, and string id.
|
|
masked_values(data, value, rtol=1e-05, atol=1e-08, copy=1, savespace=0, axes=None, attributes=None, id=None)
Constructs a variable with the given list of axes and attribute dictionary, whose mask is set at those places where
abs (data - value) < atol + rtol * abs (data).
This is a careful way of saying that those elements of the data that have value = value (to within a tolerance) are to be treated as invalid. If data is not of a floating point type, calls masked_object instead.
|
|
ones(shape, typecode='l', savespace=0, axes=None, attributes=None, id=None)
Return an array of all ones of the given length or shape.
|
|
reshape(a, newshape, axes=None, attributes=None, id=None)
Copy of a with a new shape.
|
|
resize(a, new_shape, axes=None, attributes=None, id=None)
Return a new array with the specified shape. The original array's total size can be any size.
|
|
zeros(shape, typecode='l', savespace=0, axes=None, attributes=None, id=None)
An array of all zeros of the given length or shape.
|
The following table describes the MV non-constructor functions. With the exception of argsort, all functions return a transient variable.
MV functions
|
Definition
|
|
argsort(x, axis=-1, fill_value=None)
Return a Numeric array of indices for sorting along a given axis.
|
|
asarray(data, typecode=None)
Same as cdms.createVariabledata, typecode, copy=0). This is a short way of ensuring that something is an instance of a variable of a given type before proceeding, as in
data = asarray(data)
Also see the variable astype() function.
|
|
average(a, axis=0, weights=None)
computes the average value of the non-masked elements of x along the se-lected axis. If weights is given, it must match the size and shape of x, and the value returned is:
sum(a*weights)/sum(weights)
In computing these sums, elements that correspond to those that are masked in x or weights are ignored.
|
|
choose(conditio |
Overview
