Skip to content

QuEStDataManager

Jump to bottom
Ricky Concepcion edited this page Apr 7, 2020 · 3 revisions

QuESt Data Manager

QuESt Data Manager is an application in the QuESt suite for acquiring data from open sources on the internet for use in other QuESt applications. The application contains a separate tool and UI for each data type.

The files for this application may be found in /es_gui/apps/data_manager/.

Data Manager Home

Each data acquisition tool is hosted by a Screen under management by the global QuESt Screen Manager. Each tool may implement its own screen manager as needed.

The home screen for this application changes the navigation bar. A button for each data acquisition tool is displayed.

ISO/RTO Market and Operations

This tool is for downloading market and operations data from ISO/RTO market data portals. While a lot more data is available from these sources, QuESt Data Manager only focuses on data actually used in other QuESt applications. The UI is designed accordingly.

This tool is defined in /es_gui/apps/data_manager/widgets.*.

Implementation

This tool is implemented as a TabbedPanel where each TabbedPanelItem contains a BoxLayout specific to a single ISO/RTO. The widget hierarchy can roughly be described as:

DataManagerRTOMOdataScreen:

    BoxLayout:

        [...]

        BoxLayout:

            DataManagerMarketTabbledPanel:

                TabbedPanelItem:

                    DataManagerPanelSPP:

                        [...]
                    
                TabbedPanelItem:

                    DataManagerPanelPJM:

                        [...]
                
                [...]
    
    [...]

Each panel layout is implemented differently but are designed similarly. The key design elements are:

  1. A column-oriented GridLayout to house choice selection elements. The design implies going from left to right to make selections. The number of columns depends on the number of categories of selection.
    1. A column for entering credentials, if necessary. (e.g., API key or login/password)
    2. A column for selecting the range of time for which to download data. Usually this is a range of months but it may be a range of years. Although some data portals provide sub-monthly quanta of data, the UI shall specify a granularity of months to match the time horizons of optimization.
    3. A column for specifying LMP (pricing) nodes. This can vary based on the data portal. Some portals let you specify nodes by ID. Some let you specify by some category. Depending on how the QuESt API implements downloading functions, the UI elements can vary. For example, if the function supports node IDs, a text input field can be used to accept an ID. If categories are supported, checkboxes can be used to select among those categories.
  2. Optionally, some space may be allocated for instructions/notes. For example, because of the access frequency limitations of the CAISO API, we felt it was necessary to warn the users of the five second minimum delay between queries.
  3. An output log (read-only TextInput). This is for communicating warnings, errors, and other messages to the user who may not be watching the terminal/command prompt log messages.
  4. A progress bar. This is not measured by progress in terms of bits (or size) but by quantities of files.
  5. Download button. For executing a download order based on UI selections.
  6. Cancel button. For canceling a download order.

Each panel is implemented as a separate BoxLayout-derived widget with the following class methods:

  • on_n_active_threads(): This observer method watches the n_active_threads class property. It is mainly to trigger action when the downloading threads are all finished or canceled and to report success, failure, or cancellation. It is also responsible for resetting the UI (e.g., re-enable the "download" button and disable the "cancel" button).
  • update_output_log(): Update the text in the output log.
  • increment_progress_bar(): Increase the value of the progress bar.
  • _validate_inputs(): To be called by get_inputs(). Validates the selections in the UI and raises an InputError if anything is amiss.
  • get_inputs(): Returns the selections from the UI if they pass validation.
  • execute_downloads(): Executes the download order. Retrieves selections from the UI if they pass validation; otherwise, handles any exceptions from validation and messages the user. Distributes the download requests across threads by creating batches with the module-level batch_splitter() function. Spawns a new thread for each batch, calls the relevant downloader function for each, and starts the thread.

Batch splitter

This module-level function is for splitting workloads evenly by dividing a range of months or years evenly. This is implemented by:

  1. Convert the time range into a queue of individual months or years.
  2. Compute the batch size by dividing the number of queue objects by the maximum number of threads to spawn (by default equal to 4).
  3. Create a batch by popping from the queue with chunks equal to the batch size computed in the previous step. Repeat until the queue is depleted.
  4. Returning a list of batches.

This list of batches is then used to create new threads with different workloads.

Kivy notes

Because each panel has its own UI elements and because all downloading functions are spawned in a new thread, the GUI is not blocked while download(s) are in progress. (The GUI needs the main thread to update.) In theory, downloads can simultaneously occur across the different panels. The user can leave the tool or application and the downloads will continue.

Because each of the classes are designed specifically for each ISO/RTO, they somewhat violate the MVC design pattern of QuESt.

A shortcut settings button is included on the DataManagerRTOMOdataScreen screen. It is functionally identical to the "settings" button in the navigation bar and is only there for convenience.

Commercial/Residential Load

This tool is for downloading the building load profiles from OpenEI. They are hosted in directory structures which drive the GUI design in QuESt Data Manager.

This tool is defined in /es_gui/apps/data_manager/load.*.

Implementation

Upon selecting the load profile tool, a screen for selecting "commercial" or "residential" is shown. Each of these selections will take you to a different screen (which are also managed by the global screen manager).

Both "commercial" and "residential" tools are designed identically. The selections are driven using a GridLayout with three columns. The directory structure for the data on OpenEI for each can roughly be described as:

root (commercial/residential) -> <category 1> -> <category 2> -> <category 3> (end)

For commercial building data, this is:

root -> state -> locale/measurement location -> building type

For residential building data, this is:

root -> loading (base/low/high) -> state -> locale/measurement location

Each of the three categories is selected using a RecycleView populating each column.

Upon entering the screen, a downloader function is called to query the data directory root from the online source. When successful, the first RecycleView is populated. When a selection is made, the data directory is queried to populate the second RecycleView. Likewise, this repeats for the final selection.

When the "save" button is pressed, the screen's save_load_data() method is called. This function validates the input selections then downloads the data using the appropriate downloader function.

Kivy notes

The on_enter() observer method for the Screens are used to trigger a query of the first set of options in each screen. This only needs to be done once per application run. This query is done in the main thread and blocks the GUI updates. This is done to prevent confusion; the option RecycleViews are empty until the query is complete.

Per usual, a view class for the RecycleView needs to be subclassed from the es_gui.resources.widgets.common.RecycleViewRow.

Utility Rate Structures

This tool is for acquiring U.S. utility rate structure/tariffs from the OpenEI API. Access to this API is gated by an API key that may be obtained for free after a short registration process.

This tool is defined in /es_gui/apps/data_manager/rate_structure.*.

Implementation

This tool is implemented using a wizard with a local screen manager.

The structure of the tariff database can roughly be described as:

utility -> tariff

There are static tables/spreadsheets for investor-owned and non-investor-owned utilities (IOU and non-IOU). These tables provide ways to filter utilities by, among other categories, name, state, and zip code. Based on the structure of the data, the tariff download process follows the Wizard pattern with the following order:

utility -> tariff -> verify tariff -> save

The first step is to search for the utility by name, state, or zip code. Upon executing a search, the two static tables for the utilities are downloaded and merged into a single table. Based on the search type selected, the table can then be filtered based on the search criterion (e.g., table column(s)) and search term. This can be a simple string.contains() operation or can be implemented with something like a fuzzy search.

Upon completion of the search, the list of qualifying utilities populates the utility selection RecycleView. Per usual, this RecycleView must be uniquely defined.

Selections from this RecycleView can trigger an event with the usual property observer method. When a utility is selected, the list of tariffs for the chosen utility can be queried using the utility's EIA ID obtained from the table. This list populates the next RecycleView.

When a tariff is selected, the process can move to the tariff verification/editing steps.

The energy rate schedule is represented with two schedules: one each for weekdays and weekends. These schedules are represented in the GUI using two custom widgets:

  • RateStructurePeriodTable
  • RateStructureScheduleGrid

These custom widgets are defined in the same module. The RateStructurePeriodTable is a GridLayout composed with RateStructureTableRow objects, each of which represents a rate period in the schedule. The RateStructureScheduleGrid is also a GridLayout composed with RateStructureRow objects, each of which represents either a table header or data row in a schedule. These widgets are generated and populated whenever a tariff is selected in the first screen (using a property observer method).

The schedule representation widgets are designed to be editable. While they are initialized using the data from the tariff, they can be edited before saving at user discretion.

The demand rate schedule is represented similarly as the energy rate schedule except for having an additional rate period table for flat demand rates (by month).

Other input options, such as minimum/maximum demand and net metering type, are shown in subsequent screens.

Once all inputs are defined, the tariff can be saved with a custom name.

Kivy notes

Keyboard shortcuts for the RateStructureScheduleGrid are implemented in the RateStructureTextInput class using the keyboard_on_key_down() class method.

The RateStructureScheduleGrid uses colors to differentiate among the rate periods. The RateScheduleTextInput for each entry in the schedule uses class methods to determine its background and text colors. The background_color and foreground_color properties are defined in the KV file in the widget's definition. They are bound to the widget's text property and thus update via observation of that property. They can then update dynamically using class methods.

Photovoltaic power

This tool is for acquiring photovoltaic power profiles from the PVWatts API. Access to this API is gated by an API key that may be obtained for free after a short registration process.

This tool is defined in /es_gui/apps/data_manager/pv.*.

Implementation

The inputs for the API are relatively straightforward. In addition to the API key, there are several input parameters available. This list of input parameters is governed by the Data Manager, specifically through static data:

/es_gui/apps/data_manager/_static/pvwatts_search_parameters.json

The parameter fields and descriptors are built using the static data in the PVWattsSearchParameterWidget. Default values come from the API and are displayed using hint text in the TextInput fields. These parameters are the required inputs for the API.

The API query is constructed from the input parameters per the API specs. Some static API parameters are hidden from the view and applied automatically:

pv_params['dataset'] = 'tmy3'
pv_params['radius'] = '0'
pv_params['timeframe'] = 'hourly'
pv_params['api_key'] = api_key

The dataset parameter is chosen to be TMY3 rather than the default to match the data used for the simulated building load profiles. This was advised by discussions with NREL.

Upon providing inputs, specifying a save name, and hitting the save button, the usual input validation process is performed before saving to disk.

Kivy notes

The module and array type parameters are displayed using Spinners with descriptive text for each type. These need to be translated to integers for the API query; this is done via the index method of a list.

Clone this wiki locally