Visibility Information Exchange Web System (VIEWS)
Implementation Scope of Work

(Download this document in Microsoft Word format)

Introduction

A steering committee composed of the five Regional Planning Organizations (RPOs) and the Environmental Protection Agency (EPA) has agreed to extend the current contract between CIRA and the Western Governors Association (WGA) to expand the WRAP online ambient monitoring database to a national system, creating the Visibility Information Exchange Web System (VIEWS).  The RPOs have each compiled a “wish list” of data sets and data analysis products and tools to be included in VIEWS.  These wish lists were summarized by Rich Poirot in the document “Suggested Data Input and Data Products for VIEWS” which is reproduced in Appendix A.

This scope of work summarizes and prioritizes CIRA’s tasks for migrating the WRAP Data Delivery and Analysis Web System to VIEWS and extending its data holdings and capabilities based upon the RPOs’ wish lists and CIRA’s capabilities.  The scope of work is divided into four tasks:

1)      Migrate the WRAP ambient monitoring online database website to the VIEWS website.

2)      Prepare the annual data summary of the IMPROVE and IMPROVE protocol data.

3)      Expand the online integrated database by adding data from monitoring networks throughout the United States.

4)      Enhance and add new data access and analysis tools.

Each task is described below with the priorities of the tasks and the jobs that can be accomplished in the next six to twelve months. 

The primary purpose of VIEWS is to aid the RPOs in the analysis and interpretation of air quality data to satisfy the Regional Haze regulations.  It is recognized that it is difficult to fully define one’s needs prior to conducting the data analyses.  Therefore, it is expected that the priorities of the project will change with time as the RPOs’ needs develop.  To capture the needed flexibility, the scope of work has been laid out as a set of priorities.  All items identified as a first priority have been agreed upon by the VIEWS steering committee to be highly important and will be completed in the first three to six months of this project.  Second priority items have also been identified as being desirable. Some of these items will be completed in the next twelve months.  Unless specified by the RPOs, the completion of these items will be prioritized based on the ease of accomplishment and internal needs.  The third priority items will be completed in future project years.

Migration of the WRAP website to VIEWS

The first task will be to move all of the capabilities from the WRAP ambient online database website to the VIEWS website.  The migration of the website will involve two processes.  First, the WRAP web banner will be replaced with a VIEWS banner and icon that are in development.  The second process will be to expand the spatial coverage of the current data networks in the integrated database from WRAP states to the entire United States.  The current data holdings include IMPROVE and IMPROVE protocol aerosol data, optical data and photographic spectrum series, CASTNet aerosol data and IMPROVE special studies data.  In addition, the data catalogs will be updated with links to ambient datasets relevant to all RPOs.  It is anticipated that this task will take about one month.

Annual Data Summary

Every year routine and some novel data aggregations will be conducted on the data in the VIEWS database, consisting of statistical summaries of the previous year of data.  The first data summary will be based on IMPROVE and IMPROVE protocol aerosol data beginning with the year 2000.  All calculations, where applicable, will follow the EPA guidance document for Tracking Progress Under the Regional Haze Rule.  The data summaries will consist of aerosol and light extinction spatial maps, trends, budgets, and examination of the aerosol composition and their contribution to light extinction for the 20 percent best and worst visibility days.  In addition, airmass histories will be available.  Appendix B provides a detailed listing of the planned analyses. 

Each annual data summary will consist of over 10,000 individual color graphics and making these results available in a single document or report would be burdensome on the user.  Instead, an interactive browser will be developed allowing the user to navigate through the graphics by selecting different locations, aggregation methods, metrics, parameters and time period.  The data used to generate the graphics, as well as a full description of the methodologies for generating each result, will also be available.  However, the interpretation and how the results address a particular issue are left to the user.  Therefore, the annual data summary can be thought of as a set of resources from which a user can search, browse, and draw material from to quickly generate their own data analysis report addressing specific issues.

The first priority will be to make these results and descriptions available through an interactive browser similar to the current Graphic Viewer on the WRAP website.  The second priority will be to develop searching capabilities allowing users to select groups of tables and graphics by monitoring sites, parameters and data analysis product which would be returned in a document such as MS Word or HTML.  A third priority will be to implement feedback mechanisms allowing people to comment on individual graphics.

The annual data summary is a first priority task and work has begun on this project for the WRAP States.  It is anticipated that the first report will be completed in August or September.  Because the report contains a number of different products, each product will be posted as it is completed, and the first products will be available in the initial one to two months of this project. 

Work will begin on the 2001 annual data summary when the 2001 IMPROVE and IMPROVE protocol aerosol data become available.  It is expected that this work will be completed in early 2003.  After the 2000 and 2001 data summaries have been completed, new analyses will be considered, for example, the comparison of 2000 and 2001 data analysis results.

Expansion of the VIEWS Integrated Ambient Monitoring Database

The first additions to the VIEWS integrated database will be data from the IMPROVE and CASTNet monitoring networks in non-WRAP States.  Once this is accomplished, data from networks that are not currently in the WRAP database will be added.  Table 1 lists monitoring networks in which the RPOs have expressed interest and data networks that CIRA believes would be valuable additions to the database.  Table 1, also identifies the priority for the addition of each network’s data and an estimate of the time required for adding these data.  A significant portion of the processing time will go into developing the communication protocols with the data suppliers and the database routines for automating the addition of new data in future years.  This is critical to the successful maintenance of this system in future years.

The first priority will be the addition of the AIRS PM_2.5 and speciated fine aerosol data to the database.  The time required for adding these data will depend to a large extent on our ability to work with the EPA.  We have already contacted Fred Demmick’s group at the EPA and are in discussions to see how we can work together on the transfer of these data from AIRS to VIEWS.  We have also budgeted three months of time for the addition of RPO specific datasets that have not yet been identified.  Unless otherwise specified by any RPO, the addition of the second priority networks will be prioritized by CIRA.  Several of the networks’ data are readily available via the World Wide Web.  We will add links to these data in the data catalogs in the first month of the project, regardless of the priority rating of the network.

Airmass histories have been identified as a high priority by several of the RPOs.  CIRA will add airmass histories to VIEWS in two stages.  The first stage will be the addition of four day ATAD back trajectories from every IMPROVE monitoring site from 1988 to the present.  It is recognized that ATAD is not a cutting edge trajectory model.  However, it is the only known system, with readily available wind fields, that produces unbiased summertime airmass histories in the Southwestern US in the early and mid 1990s.  The second stage will be the addition of Airmass histories generated for every IMPROVE site from 1998 onwards using the HYSPLIT or CAPITA Monte Carlo Model driven by the National Weather Service’s EDAS meteorological fields.  In conjunction with these data, an interactive tool is being developed to display trajectories for multiple locations and dates.

Data Access and Analysis tools

The primary data access and analysis tools on the WRAP website consist of the metadata browser to explore the contents of the monitoring networks and their histories that are in the WRAP integrated database, the database access tool for querying the database to return subsets of the database, and the graphic viewer for browsing preprocessed graphical and tabular data products.  

The graphic viewer will be replaced with tools developed to access the contents of the annual data analysis report previously discussed.  The annual report and the tools needed to view its contents are a first priority item and will see extensive development in the first six months of the project.  The development of the other tools has been deemed a second priority, and modest development of these tools will occur over the next twelve months.  The planned developments include enhancing the metadata browser to allow simultaneous display of information from multiple monitoring networks.  This will include identifying monitor locations on a map and the generating of location and parameter summary information from multiple networks.

The database access tool currently allows ad hoc queries to the database based upon monitoring networks, sites, measured parameters and time range.  In addition, access to calculated parameters, such as the aerosol composite components and f(rh) factors are available.  Currently, a user has to individually select monitoring sites.  The monitoring site selection capabilities will be enhanced by allowing automatic selection of groups of sites based upon selected RPO’s, States and/or proximity to Class I areas.  The time range selection will be enhanced by allowing the user to specify which years and months to select.  For instance, a user could return only summertime data for multiple years.  In addition, the output format options of the returned data will be expanded allowing the merging of monitoring site location information, e.g. latitude and longitude, into the main data table.

A new data analysis tool will be created based upon Microsoft pivot table.  This is an Excel Workbook in which both the spreadsheet and associated charts are linked to the VIEWS database via a URL.  Alternatively, subsets of the database, for example by state, RPO, or parameters of interest, can be provided as local pivot tables.  These client applications have the advantage that they are transportable and do not require access to internet connection.  The pivot tables allow one to browse and aggregate the data via drop down menus containing monitoring networks, monitoring sites, variables and time ranges, while taking advantage of the generally familiar plotting and analysis tools in Excel.  The pivot tables require Microsoft Office 2000 or later.

The RPOs have requested a number of desirable features, such as ad hoc queries of the database to return species ratios and the ability to create scatter plots.  While desirable, these features will require significant development time and will not be addressed until future years. It is hoped that the Excel pivot tables will be able to satisfy most of these immediate needs.

VIEWS Quality Assurance and Control

VIEWS will be established with a number of quality assurance and control procedures to ensure the proper addition and transformation of data into the VIEWS integrated database, the proper generation of statistical summaries of these data and the proper functioning of the data retrieval and analysis tools.  All quality control procedures will be fully described including documenting the data analysis algorithms.

RPO Training on VIEWS Contents and Tools

There is interest in conducting training exercises on the use of the VIEWS website and tools.  CIRA can conduct this training in the form of in person seminars or teleconferences.  CIRA would require one to two weeks for preparation of the training material as well as time for seminars or teleconferences. 

CIRA can hold one in-person, one-day seminar, per year.  These could be held at Colorado State University in a computer room capable of seating 30 people.  These seminars would be open to all RPOs.  This would be a good opportunity to train individuals from each RPO who could then subsequently train others in their own RPO.  Alternatively to the in-person seminar, CIRA could provide three teleconferences/year, approximately two hours in length.  All training material would be made available via the website for future reference or self study.

VIEWS will be undergoing extensive development and modifications over the first six months of this project.  Therefore, it is desirable to delay any training exercises after this initial period when the database and tools have been more fully developed.

Table 1.  Data networks to be added to the VIEWS on-line integrated database or linked to via the VIEWS data catalogs.  The list of parameters for each network is incomplete and is only intended to provide an overview of the networks contents.  Note, the processing time is in person months.

Appendix A

Suggested Data Input and Data Products for VIEWS (4/19/01 draft)

During the last (3/13/02) VIEWS Steering Committee Meeting, participants (representing all 5 RPOs):

1.      Agreed that the contract for current WRAP DIADS system could be expanded to accommodate the collective data analysis interests of all RPOs with a “modification” to the existing WRAP contract (as opposed to a major re-write of the contract),

2.      Exchanged and discussed “wish lists” (from each RPO except WRAP) of

a.      suggested data types that would be useful as VIEWS input &

b.      suggested data products that would be useful as VIEWS output.

3.      Agreed to try to summarize and prioritize these wish lists for subsequent discussion with the VIEWS contractor, to develop a modified SOW with an expanded multi-RPO focus.

This draft document attempts to summarize and prioritize the elements that might be included in a modified VIEWS contract.

A.   1^st Priority: Expand regional coverage of existing WRAP SOW

Since the other RPOs are choosing to join the existing WRAP effort, and since we all agree that our interests can be met through a modification of the existing WRAP contract, a highest priority should be given to expansion of the existing WRAP data system and SOW to a national regional scale.  The current status and near-term WRAP priorities are best summarized through review of:

The existing website

Proposed WRAP SOW

CIRA’s Response to Proposed WRAP SOW

One feature of the current website database access system is that it is focused on the IMPROVE aerosol data, including (at least placeholders for) IMPROVE protocol data (including special studies like REVEAL), CASTNET aerosol data (similar species to IMPROVE) and IMPROVE or related optical data. 

A highest priority should be to make these database extraction options fully functional for all of the above-type data for all areas of the country.

The WRAP SOW and CIRA response also refer to an “Annual Data Report” for the WRAP region.  This annual data summarization would be focused on the reconstructed extinction-related metrics that are the basis of the haze regulations (and those regulations in turn relate to metrics derived from annual aerosol data).

A highest priority should be to make these summarized metrics consistent with the most recent EPA guidance, and to develop similar annual summary reports for each RPO.

B.  New Data Input Priorities Identified by other RPOs

The original data input & output wish lists submitted bye each RPO are appended as separate attachments. Following the IMPROVE, IMPROVE protocol & CASTNET aerosol & optical data, the following appear to be the consensus priorities for additional data to include (or provide direct access to) on VIEWS:

Highest Priority New Data: PM-2.5 Mass and PM-2.5 Speciation data (from EPA AIRS database).  While these data are “available” from AIRS, they are not easily accessible in useful formats.  Ideally, a direct link to the (new) AIRS database might eventually be developed (as these data will continue to accumulate and as there are also several other AIRS data types identified as lower priorities).  However, as this is likely to require substantial time & resources, and because EPA has expressed interest in assisting the VIEWS effort, it might be most efficient in the short term if EPA could provide “batch mode” extractions (& periodic updates) to CIRA, such that a “version” of these data could be accessed directly through the VIEWS system.

Second Priority New Data: Other AIRS data, including PM-10, continuous PM-2.5 (& PM-10), gaseous precursors data.  Ability of CIRA to include these data would be dependant on ability of EPA to provide it in useful formats.  These data will be useful but are of lesser importance than the PM-2.5 mass & speciation data.

Third Priority Data: Emissions Data and Surface Meteorology. It is likely that some additional clarification is needed here to define exactly which types of emissions and surface met data would be most useful and/or most accessible.

Forth Priority Data: Assorted Complimentary Data (in no specific order of priority). Back Trajectories (for IMPROVE sites & sample days); Upper Air Meteorology data; Satellite data (or images); Deposition data (wet & dry); UV (& other) Radiation data; Special Studies data (likely varies by RPO region); Episodic Event data (wildfires, dust storms, etc.); Canadian (& Mexican) data; digital web cam data (images, including archived).

The above summary and more detailed attachments are intended as a guide for additional discussion/negotiation with the contractor – with the expectation that the contractor will have useful insights and feedback on the feasibility of adding these types of data to the VIEWS system.  Two issues that have been recognized in inter-RPO discussions are:

There’s an inherent (resource) conflict between adding (access to) additional data and developing additional data analysis products,

The contractor’s ability to “add” new data is dependent on the data generator’s ability to provide access to it.

For these reasons, it may be desirable to include some deliberate flexibility in the modified contract – combining some “required tasks” with “other tasks to the extent feasible…”

C.  New Data Output Priorities Identified by other RPOs

While there was a high degree of convergence on the suggested new data sets suggested by the RPOs, a much broader range of desired data output products was identified among the RPOs.  Consequently, the following summarized priorities are less clear, and it will be helpful for the contractor to review the more detailed original submittals from the RPOs.

·        Highest Priority Output: Annual IMPROVE Data Summary Reports (and associated data & graphics) directly associated with the Haze Regulations (and most current EPA guidance on).  This seems directly in line with the existing WRAP SOW, and would require an extension of this effort to include same for all RPO regions.

·        Second Priority Output: Intermediate Data Products and ad hoc Query Options.  This also seems more or less in line with the WRAP SOW.  Intermediate data products would include things like reconstructed species mass & reconstructed dry scattering by species that form the basis of the reconstructed extinction calculations.  Ad hoc queries would include options like species ratios, means, percentiles, aggregations (such as multi-site, monthly or seasonal means, etc.) or sorts (such as selected trace elements on only top 20% extinction days).  A special kind of useful ad hoc query (depending on whether or when its feasible to include back trajectory data for each IMPROVE site & sample day) would be an option to request trajectories that relate to any specific aerosol data selections.

·        Third Priority Output: Integrated Data Sets and/or alternative Formats These concepts are inter-related, in the sense that an integrated data set (merge of 2 or more data sets) might require formatting for a specific application – or conversely if data are provided in formats compatible with specific applications, they will be more amenable to integration and comparison.  An example might be adding optional columns indicating the latitude and longitude of each data point extracted from the on-line data database.  This might allow resulting delimited ASCII data to be entered directly to spatial analysis tools like ArcView or (compiled for) Voyager.  Examples of logically useful merged datasets might be: IMPROVE + CASTNET + EPA STN speciated aerosol data; reconstructed extinction + measured extinction (aggregated to common time steps); aerosol sulfate + SO2 emissions (aggregated to common time and space scales).

·        Forth Priority Output: Assorted Complimentary Data (in no specific order of priority).  Statistical Data Analysis tools; Metadata Search options; Scatter Plot output option; Haze Metric-related calculations for non-IMPROVE sites (like CASTNET & EPA STN); “Actual f(RH)” calculator; “Real-Time VIEWS”; Gridded data (and/or GIS shape files); On-Line Tutorial (or Hands-On Training); User Comment/Feedback options.

As with the suggested new input data, it’s recognized that this list of desired output products is extensive, adding new output options conflicts with resources available to refine existing ones, and that some flexible contract language may be useful. Some high priority options might be best deferred to future years.

Appendix B

Planned Content for the First Annual Data Summary

This appendix list all of the data aggregations to be conducted for the first VIEWS annual data summary.  These data aggregations will be conducted for all IMPROVE and IMPROVE protocol monitoring sites over a one year time period with the first report covering the 2000 aerosol data.  This effort will result in over 10,000 graphics for each data summary.

VIEWS has expressed interest in also conducting these data aggregations for a five year time period.  This will be done for the 1996-2000 time period, provided time permits.  New data aggregations will be considered after the 2000 and 2001 data summaries are available, for example, the 2000 results could be compared with the 2001 results.   

Spatial analyses

Colored contour or isopleth maps will be created for fine and coarse mass, the fine mass composite components and their contribution to light extinction as well as total light extinction.  The fine mass composite components consist of ammonium sulfate, organics, soil, ammonium nitrates and elemental carbon.  These contour maps will be created for absolute and fractional contributions of the annual and seasonal averages and average of the best and worst 20% visibility days.  The aggregations will follow the methodologies specified in the EPA guidance document for Tracking Progress Under the Regional Haze Rule.

There will be approximately 30 maps in total.

Aerosol and Light Extinction Budgets

Time charts, bar charts and pie charts will be used to display the aerosol and light extinction budgets for each IMPROVE and Protocol monitoring site.  The fine mass budgets will consist of the fine mass composite aerosol components while the light extinction budget will include the contributions to light extinction from the fine mass composite aerosol components, coarse mass and light absorption.

Aerosol and light extinction budgets of the absolute and relative contributions of the annual and seasonal averages and average of the best and worst 20% visibility days will be calculated.  The aggregations will follow the methodologies specified in the EPA guidance document for Tracking Progress Under the Regional Haze Rule.

There will be approximately 24 graphs/site.

Long Term Trends

Long term trend plots will be created for the average of the worst, middle and best 20% of the visibility days of the fine and coarse mass, fine mass composite aerosol components and their contribution to light extinction, and the reconstructed light extinction and deciview.  The aggregations will follow the methodologies specified in the EPA guidance document for Tracking Progress Under the Regional Haze Rule.

There will be approximately 28 graphs/site.

Aerosol and Light Extinction budgets for the Individual Best and Worst 20% of the Visibility Days

Aerosol and light extinction budgets will be generated for each day comprising the best and worst 20% of the visibility days for each IMPROVE and protocol monitoring site.  The budgets will be for the year 2000.  The aggregations will follow the methodologies specified in the EPA guidance document for Tracking Progress Under the Regional Haze Rule.

There will be 2 graphs/site.

Airmass Histories

Airmass histories using HYSPLIT or the CAPITA Monte Carlo model driven by the EDAS wind fields will be generated for the 20% best and worst days at each IMPROVE and IMPROVE protocol site for 2000. The airmass histories will include the airmass location, precipitation, temperature and humidity experienced by the air parcels in route to the receptor.   Note, these airmass histories are in addition to the airmass history database to be developed for VIEWS, as discussed in the section “Expansion of the VIEWS Integrated Ambient Monitoring Database.”

There will be approximately 40 plots/site.