Skip to main content

VisionEval User Guide

8 Validation and Troubleshooting

Each model run will produce a time stamped Log.txt file. This log file is the first thing that the user should reference in troubleshooting errors. The log file should point to which input files have errors and help to define the error. If a model is not producing a log file, that indicates a more deep-rooted issue related to the model set-up, installation, or one or more packages.

Other common errors users should watch out for include:

  • Files having data for years other than model run years will generate an error, files can only have data for model run years
  • Consistent geography names across input files
  • Input files in different modules need to be consistent
    • Zero land area for an azone location type when there are non-zero values for households in that azone location type
    • Zero land area for an azone location type when there are non-zero values for employment in that azone location type

8.1 Validation

This section summarizes additional detail on the validation of VisionEval models with key considerations by concept. Each concept will highlight its respective model inputs and assumptions that can be changed to better match observed local patterns and trends. These outputs covered are only a handful that can be used to validate the VE model.

8.1.1 Household Synthesis and Land Use Validation

Two key metrics to validate are population and income due to the influential nature of these metrics on model results. Some additional considerations are as follows:

  • The choice of geographies used in VisionEval can influence validation results. For example, if economic conditions or driver licensing rates vary significantly across the modeled area it might be a good idea to define Azones to reflect those differences.
  • The VESimHouseholds package processes PUMS data to derive parameters for several of its sub-modules. The default PUMS files in the inst/extdata folder from Oregon should be replaced with data for the area they are modeling. This is done by simply replacing the PUMS data in the inst/extdata folder with local data, which are then processed by several modules as part of a normal model run. This requires rebuilding the VESimHousehold package, more information can be found in the Module Build Process chapter.
  • The average household size (AveHhSize) and proportion of one-person households (Prop1PerHh) can be set in the azone_hhsize_targets.csv file.
  • Care should be taken to match the real dollar amount of azone_per_cap_inc.csv and other files with dollar values to the year specified in the file to account for inflation.
  • The relative employment rate by age group by Azone can be specified to match observed differences across a metropolitan area or levels at various points of economic cycles. Lower employment rates, especially in certain age groups, should be reflected in azone_relative_employment.csv. The relative employment rate is relative to the average employment rate for the worker group in the PUMS data (e.g., a value of 0.5 would be entered if the employment rate for 20-29 age group in one Azone was half the employment rate for persons in that age group in the metropolitan area).
  • The VELandUse package also uses the PUMS data, which can be further adjusted in two ways:
    • The mix of single family versus multi-family households will reflect local patterns if PUMS data for the modeled region are used instead of the default data from Oregon. See the [ insert link to the build process PUMS case study]
    • The proportion of households residing in mixed-use neighborhoods within each Bzone can optionally be set in the bzone_urban-mixed-use_prop.csv file. Such adjustments are subjective, as the definition of “urban mixed-use” neighborhoods derived from the Claritas data in the NHTS are imprecise.

8.1.2 Household Travel Behavior Validation

One key metric to validate for household travel behavior is household DVMT. Because VisionEval models do not have a network, users should prioritize validating household DVMT using NHTS or local household travel survey data. Users can also use Highway Statistics reports or imputed from annual HPMS estimates to validate at higher levels of geography, such as the state or metropolitan area level.

  • Care should be taken in choosing the validation targets to match model predictions. VE household travel modules predict household travel regardless of where the travel occurs within the modeled region. Therefore the results should be compared with household survey data where possible which provide similarly defined estimates.
  • Care should similarly be taken in comparing the model results with daily roadway VMT data, such as those reported in Highway Statistics reports or imputed from annual HPMS estimates. If such comparisons are made the model estimates of roadway DVMT calculated by the CalculateRoadDvmt module should be used for comparison (rather than HPMS data that is defined as all vehicles but only those miles on roadways within a specific geography).
  • If the modeled road DVMT estimates match for the base year but not for prior years the modeled household DVMT trends should be checked against road DVMT trends. In particular, check whether reduction of DVMT (or reduction of DVMT growth rate) observed during the Great Recession and increase in DVMT (or increase in the DVMT growth rate) afterwards are reflected in the modeled household DVMT. If not, the values in the following files should be checked and adjusted if warranted:
  • There are few opportunities to adjust the parameters in the VEHouseholdTravel package, as most are derived from the NHTS and PUMS input data. Note that the optionally specified driver licensing rates described above can substantially affect daily household VMT. It is possible to specify the percentage of household SOV travel diverted to bicycles in the azone_prop_sov_dvmt_diverted.csv file to better match observed local values.

8.1.3 Vehicles and Fuels Validation

Another important metric to validate is vehicle ownership, which has a strong correlation with household DVMT.

  • If modeled DVMT matches validation data but fuel consumption does not (e.g. light-duty vehicle fuel consumption reported in Highway Statistics), the values in ldv_powertrain_characteristics.csv for current and past years can be adjusted to achieve a match. Fuel consumption may not match for several reasons such as:
    • The definition of crossover vehicles as light trucks vs. autos used in the vehicle type model does not match the average fuel consumption characteristics in the ldv_powertrain_characteristics.csv file.
    • The average fuel consumption characteristics in ldv_powertrain_characteristics.csv do not represent real world fuel economy.
  • Most of the parameters in the VEHouseholdVehicles package are self-calibrating. However, the relative driver licensing rate by age group can be coded in the region_hh_driver_adjust_prop.csv file and should be used to account for the reduction in driver licensing rates among young or elderly drivers.
  • Users can also use the azone_hh_ave_veh_per_driver.csv to reduce or increase vehicle ownership at the Azone level. It should be noted that this file inherently reduces the sensitivity of the AdjustVehicleOwnership module.
  • If validation data are available for commercial service vehicles, heavy trucks, and public transit vehicles changes can be made to the respective powertrain characteristics files for those vehicle types to match observed values.
  • Rebuilding the VEPowertrainsAndFuels package is good practice. The default data inputs in the VEPowertrainsAndFuels package substantially affect modeled fuel consumption and vehicle emissions rates. These default inputs are contained in the inst/extdata folder of the source package. Note that the package needs to be built (installed) from the source package after adjustments have been made in order for the changes to have effect. more information can be found in the Module Build Process chapter.

8.1.4 Congestion and Roadway Travel Validation

  • The VETravelPerformance package is self-calibrating. However, the user must provide several estimates used as constraints during that process:
    • Estimates of urbanized area light-duty vehicle and heavy truck VMT (UrbanLdvDvmt and UrbanHvyTrkDvmt, respectively) must be coded in the marea_base_year_dvmt.csv input file.
  • The user must also provide a regional estimate of heavy truck VMT (HvyTrkDvmt) in the region_base_year_dvmt.csv that is consistent with the urbanized area heavy truck VMT estimates.
  • The user should check the basis used for estimating commercial service VMT (ComSvcDvmtGrowthBasis) and heavy truck VMT (HvyTrkDvmtGrowthBasis).

8.2 Additional Customizations

The VESimHouseholds and PowertrainsAndFuels packages are the two that should be prioritized for re-estimating with local data. Users do have the option to customize or re-estimate other model packages based on local data. Some additional packages with built-in estimation scripts are described below.

Note: For a deeper dive into how to customize packages and the various data that is available for local estimation, users should reference the Estimation in VisionEval and Module Build Process chapters.

  • VETravelPerformance: The LoadDefaultRoadDvmtValues script pulls in datasets from the 2010 Highway Statistics reports are used to calculate state and urbanized area travel statistics as described below. A data from the Transportation Energy Databook (Edition 31) are used to calculate the ratio of commercial service vehicle DVMT with household DVMT. These datasets are in the “inst/extdata” folder of the package. Documentation for these datasets are included. Advanced users may update the datasets if desired.

  • BudgetHouseholdDvmt: The CES data used to estimate the BudgetHouseholdDvmt model are included in inst/extdata folder of the source package in the ces_vehicle_op-cost.csv with documentation in ces_vehicle_op-cost.txt. The ces.R R script file contains the code used to download the raw CES dataset from the BLS website and process it to produce the dataset in the ces_vehicle_op-cost.csv file. CES data for the years 2003 to 2015 are used in model estimation. 2003 being the first year that the BLS included income subcategories for incomes greater than $70,000. 2015 being the last year of complete data when the model was estimated.

  • VETravelPerformance & CalculateVehicleOperatingCost: Vehicle maintenance, repair, and tire costs as a function of the vehicle age are calculated based on data from the American Automobile Association (AAA) and the Bureau of Labor Statistics (BLS). AAA publishes reports yearly on the cost of vehicle use by vehicle type over the first 5 years of the vehicle’s life. The 2017 report, a copy of which is included as the ‘17-0013_Your-Driving-Costs-Brochure-2017-FNL-CX-1.pdf’ file in the inst/extdata/sources directory of this package, is used to calculate baseline MRT cost by vehicle type. Data from a BLS report, “Beyond the Numbers, Prices and Spending, Americans’ Aging Autos, BLS, May 2014, Vol.3/No.9”, are used to establish the relationship between MRT cost and vehicle age. A copy of the report is included as the ‘americans-aging-autos.pdf’ file in the inst/extdata/sources directory of this package. This report includes estimates of average MRT cost by vehicle age category for all household vehicles. The MRT costs by vehicle type and age are calculated as the outer product of the AAA costs by vehicle type and the BLS ratio of MRT cost by vehicle age. Since the BLS data don’t distinguish between vehicle types, it is assumed that the effect of age on MRT expenses is the same for all vehicle types.

  • VETravelPerformance & CalculateVehicleOperatingCost: Default carbon cost values are from “Technical Support Document: Technical Update of the Social Cost of Carbon for Regulatory Impact Analysis Under Executive Order 12866, Interagency Working Group on Social Cost of Greenhouse Gases, United States Government, August 2016”. A copy of the report is included as the ‘sc_co2_tsd_august_2016.pdf’ file in the inst/extdata directory of this package. Carbon costs are estimated by year and assumed discount rate scenarios: 5%, 3%, 2.5%. In addition, they are calculated for a lower probability but higher impact scenario. The default carbon costs used in the model are the values listed for the 3% discount rate. Non-carbon social costs for other social costs are derived from an white paper prepared for ODOT to support the development of ODOT’s statewide transportation strategy for reducing greenhouse gas emissions from the transportation sector. This paper is included as the ‘STS_White_Paper_on_External_Costs_9-21-2011.pdf’ file in the inst/extdata directory of this package. The included social cost categories are air pollution, other resource pollution, energy security, safety, and noise.